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I.  INTRODUCTION 


This  research  develops  an  instrument  that  can  be  used  to  measure  the 
perceived  value  of  the  Manpower  Systems  Analysis  (MSA)  curriculum  in  the 
Graduate  School  of  Business  and  Public  Policy  at  the  Naval  Postgraduate 
School.  In  order  to  understand  the  MSA  curriculum,  first  one  needs  to  examine 
the  history  of  the  Navy’s  Human  Resource  Officer  Community,  one  of  the  main 
stakeholders  in  the  MSA  graduate  education  program.  It  is  also  important  to  look 
at  the  detailing  process  for  officers  holding  a  graduate  degree,  and  some  general 
background  on  graduate  education  in  the  military.  Finally,  this  chapter  discusses 
the  goals  of  the  surveys  and  outlines  the  research  questions  that  this  study 
addresses. 

A.  EVOLUTION  OF  THE  NAVY’S  HUMAN  RESOURCE  COMMUNITY 

The  Navy’s  Human  Resource  Officer  Community  is  fairly  new  in  its  current 
form.  It  has  gone  through  a  series  of  changes,  redesignations,  and  restructuring. 
In  1972,  in  reaction  to  the  Combat  Exclusion  Law,  which  barred  women  from 
serving  in  most  Navy  billets,  the  surface,  submarine,  and  aviation  warfare 
Unrestricted  Line  (URL)  communities  were  created.  (FitzPatrick,  1998,  p.1 1)  The 
officers  who  remained  were  primarily  women  and  a  small  number  of  male  attrites 
from  other  officer  communities  (due  to  poor  health  or  performance).  (FitzPatrick, 
1998,  p.1 1)  This  group  of  non-warriors  was  called  the  General  Unrestricted  Line 
(GenURL)  community  (designator  1100).  This  community  consisted  of  the 
Navy’s  fleet  support  officers,  which  included  manpower  and  personnel 
specialists.  With  the  establishment  of  the  community,  the  Navy’s  first  human 
resource  officers  were  created  with  the  following  mission  statement: 

The  mission  of  the  General  Unrestricted  Line  Community  is  to  provide 
the  Navy  with  a  community  of  Officers  of  proven  leadership,  shore 
management  and  subspecialty  expertise  who  manage  the  increasingly 
complex  fleet  support  establishment  in  direct  support  of  the  Navy’s  war 
fighting  mission.  (Barber,  2003,  p.9) 
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Even  though  the  community  was  formally  recognized  and  had  an  assigned 
mission  statement,  it  still  experienced  problems.  The  community  was  seen  by 
many  as  just  a  place  where  women  officers  could  serve,  with  no  real  purpose  or 
value  to  the  Fleet.  (Barber,  2003,  p.  9) 

The  community  yet  again  transformed  in  1994,  with  the  establishment  of 
the  Fleet  Support  Officer  (FSO)  Community.  Each  of  the  1 100  GenURL,  officers 
was  re-designated  into  the  1700  community.  The  FSOs  had  the  three  “core 
competencies”  of  Logistic  Support,  Space  and  Electronic  Warfare,  and 
Manpower,  Personnel  and  Training  (MPT).  (Barber,  2003,  p.10)  This  conversion 
only  lasted  for  a  short  time  because  in  1999,  all  1700  FSO  officers  were 
converted  back  to  GenURL,  1100  officers,  again  in  part  due  to  community 
credibility  issues.  (Barber,  2003,  pp. 14-15) 

In  2001,  the  Human  Resource  Officer  Community  (NHROC),  as  it  is  today, 
was  established.  Each  of  the  329  Human  Resources  (HR)  Officers,  who  started 
the  1200  community  was  selected  via  a  Special  Transition  Board.  This 
community  was  created  “to  meet  the  growing  demands  for  specialized  skills  in 
human  resource  strategy  and  analysis.”  1  (Barber,  2003,  p.  1 7)  Although  the 
community  is  still  battling  with  the  URL  communities  to  be  recognized  as  a 
legitimate  asset  to  the  fleet,  the  community  mission  and  career  progression  is 
less  ambiguous.  Today  the  NHROC  community: 

Provides  life-cycle  management  of  Navy  people  through  requirements 
determination,  force  shaping,  recruitment  and  selection,  inventory 
management,  and  workforce  training  and  development.  (NPS  2006) 

The  Human  Resource  Community  is  an  active  participant  in  supporting  the  war 
effort.  According  to  RADM  Crisp,  the  highest  ranking  HR  officer,  the  1200 
officers’  expertise  have  been  invaluable  during  the  Global  War  on  Terrorism 
(GWOT),  the  Quadrennial  Defense  Reviews  (QDR),  Manpower,  Personnel, 
Training  and  Education  (MPT&E)  alignment,  and  development  of  the  Human 

1  For  further  information  on  the  history  of  the  1200  Navy  Human  Resources  Community  refer 
to  NPS  theses  written  by  Barber  Jr.,  Harry  C.  (2003)  and  FitzPatrick,  Erick  L.  (1998). 
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Capital  Strategy,  and  Task  Force  Total  Force  (TFTF).  (Crisp  2005)  With  Navy 
senior  leadership  turning  to  HR  officers  for  guidance  in  HR-related  subjects,  and 
the  HR  field  changing  so  quickly,  the  importance  of  graduate  education  continues 
to  increase  in  order  to  have  specialists  who  are  knowledgeable  about  the  latest 
HR  practices. 

B.  BACKGROUND  OF  GRADUATE  EDUCATION  IN  THE  NAVY 

In  1990,  the  Department  of  Defense  (DoD)  issued  a  “Policy  on  Graduate 
Education  for  Military  Officers.”  This  directive  established  the  importance  of 
graduate  education  for  military  officers  and  provided  a  DoD-wide  policy  for 
funding  of  higher  education.  The  directive  was  written  in  an  effort  to  “Raise  the 
levels  of  individual  military  officer  professionalism  and  technical  competence  so 
that  those  officers  more  effectively  perform  their  required  duties  and 
responsibilities.”  (DODD  1322.10)  After  the  written  directive  was  issued,  each  of 
the  military  services  placed  an  increasing  emphasis  on  graduate  education. 
Evidence  of  this  importance  can  be  seen  by  each  of  the  services’  issuance  of 
their  own  graduate  education  instructions. 

Following  the  guidance  on  graduate  education  outlined  by  DoD,  the 
Navy’s  instruction  identifies  the  need  for  officers  with  specific  subspecialty 
education  gained  through  graduate  education.  The  instruction  also  addresses 
the  recruiting  and  retention  benefits  that  are  created  by  funded  graduate 
education.  Graduate  education  can  be  achieved  through  a  series  of  programs  in 
the  Navy.  Officers  can  be  selected  for  a  fully  funded  gradate  program,  or  can 
pursue  further  education  through  different  partially  funded  or  fully  funded 
programs  including:  the  Navy  Campus,  Tuition  Assistance  (TA),  the  Montgomery 
Gl  Bill,  or  a  Graduate  Education  Voucher  (GEV). 

1.  NHROC  Graduate  Education 

Earning  a  master’s  degree  early  is  part  of  the  career  progression  of  the 
Navy  HR  Officer.  A  graduate  education  is  important  for  promotion  and  screening 
for  milestone  billets  throughout  the  Navy,  and  in  the  1200  community.  According 
to  the  HR  Community  Manager  and  the  Head  Detailer,  extensive  knowledge  is 
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needed  in  the  HR  Community  in  the  following  HR  related  areas:  manpower; 
operations  analysis;  education/training;  financial  management;  human 
performance  technology;  and  human  resources  support  systems.  (USN  2005b) 
The  value  of  a  postgraduate  education  for  the  HR  community  is  proven  by 
examining  the  2005  promotion  and  Major  Command  board  results.  All  of  the 
Captains  (0-6)  selected  for  Major  Command,  and  90  percent  of  those  selected 
for  Commander  (0-5)  had  a  master’s  degree.  Of  those  selected  for  Lieutenant 
Commander  (0-4),  only  55  percent  had  a  master’s  but  most  others  had  made 
significant  progress  towards  an  “HR/Management”-related  master’s.  (USN 
2005b)  The  career  progression  of  the  HR  Officer  typically  allows  for  earning  a 
master’s  degree  as  a  Lieutenant  (0-3)  or  as  a  junior  0-4.  This  career 
progression  chart  can  be  seen  in  Figure  1  below.  A  majority  of  the  1200 
community  is  accessed  through  lateral  transfers  from  Unrestricted  Line 
communities,  so  there  is  limited  opportunity  for  an  officer  to  have  a  degree  prior 
to  the  0-4  board,  accounting  for  the  lower  number  for  0-4s  with  a  graduate 
degree. 
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Figure  1 :  Navy  HR  Officer  Career  Progression  (From:  USN  2005b). 


2.  The  Naval  Postgraduate  School  and  the  MSA  Curriculum 

One  of  the  Navy’s  main  graduate  education  programs  for  HR  officers  is 
offered  through  the  Naval  Postgraduate  School  (NPS).  Over  forty  different 
master’s  degrees  are  available  at  NPS  in  the  schools  of  Business  and  Public 
Policy,  Engineering  and  Applied  Sciences,  Operational  and  Information 
Sciences,  and  International  Graduate  Studies.  The  Manpower  Systems  Analysis 
(MSA)  Curriculum  is  offered  as  part  of  the  School  of  Business  and  Public  Policy. 
This  curriculum  is  a  21-month  program  addressing  Manpower,  Personnel, 
Training  and  Education  (MPT&E)  issues  as  they  relate  to  DoD  and  the 
Department  of  the  Navy  (DoN).  Officers  enrolled  in  the  MSA  curriculum  are  from 
various  services  and  countries.  However,  many  of  the  Naval  Officers  enrolled 
are  from  the  Navy’s  Human  Resource  Community  (officers  with  a  1200 
designator).  With  the  Navy’s  recent  discussions  on  downsizing  and 
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reorganization,  expertise  in  manpower  and  human  resource-related  issues  is 
highly  desired.  Chief  of  Naval  Operations  (CNO),  Admiral  Mike  Mullen,  stated 
one  of  his  top  priorities  for  2006  is  to  “Develop  21st  Century  leaders.”  One  of  his 
desired  effects  is  to  have  “A  transformed  manpower  and  personnel  system 
[which]  delivers  an  improved  quality  of  service  that  is  more  responsive  to  the  men 
and  women  serving  the  Navy.”  (Mullen,  2006,  pp.3-4)  In  order  to  develop  these 
“21st  Century”  leaders  who  can  transform  the  manpower  and  personnel  system, 
education  is  necessary  and  the  MSA  curriculum’s  current  relevance  has  grown. 

A  Naval  graduate  from  the  MSA  curriculum  is  assigned  the  3130P 
(Manpower  Systems  Analysis)-subspecialty  code.  The  P  indicates  the  person 
received  graduate  education  in  the  subject.  The  NPS  MSA  curriculum  is  the  only 
source  for  an  officer  to  obtain  the  3130-subspecialty.  All  Marine  Corps  MSA 
graduates  receive  the  9640-Military  Occupational  Specialty  (MOS),  Manpower 
Management  Officer.  Although  non-DoD  students  attend  the  curriculum,  this 
study  looks  at  only  Navy  students  in  the  MSA  Curriculum. 

3.  Other  Military  and  Civilian  Institutions 

In  addition  to  the  Naval  Postgraduate  School,  Navy  funded  Military 
Graduate  Education  programs  are  offered  at  the  Air  Force  Institute  of  Technology 
(AFIT),  National  Defense  University,  the  Defense  Intelligence  College  (DIC),  the 
Air  Command  and  Staff  College,  and  the  Naval  War  College.  There  are  also 
special  graduate  programs  offered  at  several  civilian  institutions  including  the 
Massachusetts  Institute  of  Technology  (MIT),  Harvard,  and  Duke.  The  civilian 
universities  that  are  approved  by  the  Chief  of  Naval  Operations  vary  by  course  of 
study. 

There  is  no  civilian  or  military  university  that  offers  a  master’s  that  can 
duplicate  the  MSA  curriculum.  There  is  an  Education  and  Training  master’s  that 
is  offered  at  six  different  universities  across  the  country.  However  none  of  these 
curricula  result  in  the  same  Navy  subspecialty  code  as  the  MSA  curriculum.2 

2  As  outlined  in  OPNAV  Notice  1520  (Oct  1996),  the  six  universities  that  offer  the  Education 
and  Training  masters  are  George  Mason  University,  University  of  Memphis,  Old  Dominion 
University,  San  Diego  State  University,  University  of  Rhode  Island,  University  of  West  Florida. 
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(USN  1996)  The  only  way  to  receive  the  3130X-subspecialty  code  is  by 
completing  the  MSA  curriculum  at  the  Naval  Postgraduate  School.  The 
Education  and  Training  master’s  degree  offered  at  the  civilian  universities  also 
does  not  address  the  topic  from  the  same  military  aspect  as  the  education  given 
at  the  military  facilities.  An  HR-related  master’s  is  preferred  in  promotion  and 
milestone  selection  boards  for  the  Navy’s  HR  community.  In  addition  to  the 
Education  and  Training  Masters,  HR  officers  can  also  work  towards  masters’ 
degrees  in  the  following  HR  related  areas:  business  administration; 

economics/finance;  educational  technology;  human  resource  management; 
human  systems  engineering;  operations  management;  organizational 
management;  and  organizational  development.  The  value  of  the  military- 
specific  education  has  not  been  established  but  is  one  factor  to  consider  when 
the  Navy  is  selecting  an  education  source  for  officers. 

C.  BILLETING  PROCESS 

According  to  both  the  DoD  Directive  and  the  OPNAV  Instruction  on 
Graduate  Education,  an  officer  who  receives  a  fully  or  partially  funded  graduate 
degree  must  conduct  a  “payback  tour”  following  the  education.  The  eligibility  of  a 
payback  tour  is  considered  a  “validated  position”  that  requires  the  subspecialty 
received.  The  payback  must  be  completed  within  two  tours  following  the 
education;  however,  a  quick  payback  is  preferred.  (DoD  2004,  p.3  and  USN 
1991b,  p.2)  For  Navy  personnel,  any  exceptions  to  the  payback  requirement 
must  be  approved  by  the  Chief  of  Naval  Personnel. 

1.  Navy  Billeting  Process 

Since  the  policy  on  utilization  of  graduate  education  is  so  strict,  it  is 

surprising  that  the  Navy’s  billet  process  is  not  more  stream-lined.  With  only  a  few 

exceptions,  the  Navy’s  officer  billeting  process  is  conducted  on  a  by-tour  basis. 

This  means  the  person  is  only  detailed  to  his  or  her  next  command,  with  no 

discussion  of  follow-on  billets.  This  can  prove  to  be  especially  difficult  once  a 

subspecialty  is  acquired.  With  the  Navy’s  policy  on  subspecialty  utilization,  using 

the  MSA  curriculum  as  an  example,  the  graduate  should  ideally  be  detailed  to  a 

3130P-coded  billet.  This  follow-on  billet,  requiring  the  MSA  graduate  degree 
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would  serve  as  the  “payback  tour,”  earning  the  graduate  the  3130Q-subspecialty 
upon  completion.  The  “Q”  indicates  that  the  person  not  only  received  graduate 
education  in  the  subject,  but  is  also  a  “proven  subspecialist”  with  professional 
experience  in  the  field.  An  officer  with  significant  on-the-job  experience  and  the 
MSA  master’s  would  receive  a  3130R-subspecialty  code.  (USN  2005a) 
Although,  ensuring  each  member  completes  his  or  her  payback  tour  is  not  as 
difficult  in  the  HR  community  as  it  is  in  the  Unrestricted  Line  Community,  it  is  still 
an  additional  billeting  consideration. 

2.  Marine  Corps  Special  Education  program 

The  Marine  Corps  detailing  process  following  graduate  education  is 
different  from  the  Navy’s.  Instead  of  finding  billets  for  the  graduate  after 
graduation  (or  close  to  graduation),  the  Marine  Corps  has  a  program  established 
specifically  to  track  and  manage  graduate  education  and  education  utilization. 
The  purpose  of  this  program,  entitled  the  Special  Education  Program  (SEP),  is  to 
coordinate  several  hundred  billets  requiring  postgraduate  education  with 
graduate  education  curricula  and  officer  placement  in  those  curricula.  (USMC 
2003).  The  program  validates  the  graduate  education  billets  and  most  students 
know  where  they  will  be  going  for  their  36-month  payback  tour  when  they  start 
their  graduate  programs.  Most  of  the  funded  postgraduate  education  in  the 
Marine  Corps  is  managed  by  the  SEP. 

D.  NEED  FOR  SURVEY 

Throughout  the  corporate  world,  companies  are  trying  to  determine  how  to 
measure  human  capital.  Human  capital  development  can  occur  through  on-the- 
job  experience,  formal  education,  or  company-specific  training.  The  value  in  on- 
the-job  training  and  company-specific  training  can  usually  be  easily  seen  if  the 
organization  decides  to  measure  it.  What  is  not  as  straightforward  is  the 
economic  return  an  organization  gets  from  funding  formal  education  that  is 
transferable  to  other  companies.  Is  the  amount  of  return  to  the  company  worth 
sending  an  employee  to  receive  higher  education?  How  can  the  organization 
measure  the  worth  of  a  person  with  and  without  graduate  education? 
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The  Navy  is  just  like  any  other  company;  it  too,  needs  to  determine  if  the 
investment  in  education  has  a  sufficient  economic  return.  Whether  the  return  is 
in  the  form  of  increased  productivity  or  efficiency  of  the  graduate,  or  merely  an 
increase  in  retention,  the  magnitude  of  the  return  needs  to  be  measured.  When 
it  comes  to  NPS  specifically,  the  Resource  Sponsors  need  to  know  if  each 
curriculum  offers  what  is  necessary  to  prepare  officers  for  their  payback  tours.  It 
is  important  to  determine  if  an  education  at  NPS,  in  this  case  the  MSA 
curriculum,  generates  value  to  the  Navy.  The  goal  of  this  thesis  is  to  develop  two 
surveys  that  can  be  used  to  measure  the  perceived  value  of  the  MSA  curriculum 
to  corporate  Navy. 

As  per  the  requirements  established  by  the  Navy  Instruction,  OPNAVINST 
5450.21  OB,  the  curriculum  is  reviewed  biannually  and  changes  are  made  to  keep 
the  curriculum  current  to  the  needs  of  the  Navy.  Changes  in  the  past  have 
included  changing  the  MSA  degree  for  most  Navy  students  to  a  Master’s  in 
Business  Administration  (MBA)  instead  of  a  Master’s  in  Science  (MS).  In  the 
most  recent  curriculum  review,  however,  the  sponsor  requested  the  degree 
revert  to  an  MS.  In  addition,  the  curriculum  review  is  conducted  to  ensure  that 
the  MSA  curriculum  maintains  its  relevance  and  is  effective  in  preparing  Navy 
graduates  for  3130-subspecialty  billets. 

1.  Sponsor  of  MSA  Curriculum 

A  Resource  Sponsor  is  responsible  for  the  program’s  content  and 
execution,  and  ultimately  defending  the  program  in  the  budget  process.  The 
Navy  defines  a  resource  sponsor  as  the  following: 

The  OPO  [OPNAV  Principle  Official]  responsible  for  an  identifiable 
aggregation  of  resources  which  constitute  inputs  to  warfare  and 
supporting  tasks.  The  span  of  responsibility  includes  interrelated 
programs  or  parts  of  programs  in  several  mission  areas.  (USN  1998, 
Appendix  B) 

The  Navy  Human  Resource  community’s  resource  sponsor  is  the 
Manpower  and  Personnel  organization  (N1).  The  mission  of  the  N1  organization 
is  to  “support  the  needs  of  the  Navy  by  providing  the  Fleet  with  the  right  person  in 


9 


the  right  place  at  the  right  time.”  (USN  2006)  A  subset  of  the  Manpower  and 
Personnel  Organization,  the  Strategic  Planning  and  Analysis  Directorate  (N14),  is 
the  resource  sponsor  for  the  MSA  curriculum.  As  the  resource  sponsor,  N14  is 
directly  interested  in  maintaining  the  validity  of  the  MSA  curriculum  and  adapting 
the  courses  to  meet  the  changing  needs  of  the  Navy.  As  a  parent  organization  of 
N14,  N1  is  ultimately  also  a  key  shareholder  in  the  review  process.  As  per  Navy 
instruction,  N1  is  required  to  review  and  validate  each  subspecialty  every  two 
years.  (USN  1998,  p.8-1)  As  a  result,  NPS  is  also  required  to  conduct 
subspecialty  curriculum  reviews  biennially,  submitting  written  reports  via  the  N14 
organization  to  the  CNO.  (USN  1 991 ) 

2.  Current  Curriculum  Review  Process 

The  review  process  consists  of  key  curriculum  advisors  discussing 
possible  changes,  directives  from  the  administration  (in  the  School  of  Business 
and  NPS-wide  policies),  and  any  requested  changes  from  the  resource  sponsor. 
In  addition,  the  Marine  Corps  outlines  specific  requirements  for  Marine  Corps 
students  in  the  MSA  curriculum.  Currently,  the  only  student  and  graduate 
feedback  that  is  used  in  the  curriculum  review  process  consists  of  Student 
Opinion  Forms  (SOFs)  and  exit  surveys  of  graduates.  Besides  the  exit  surveys 
conducted,  no  feedback  loop  exists  to  capture  lessons  learned  from  graduates  in 
follow-on  MPT&E  billets  or  from  their  supervisors. 

3.  Goal  of  Survey 

This  thesis  proposes  that  an  external  feedback  loop  will  be  created  using 
a  survey  given  to  both  graduates  and  their  immediate  supervisors.  The  surveys 
are  a  vehicle  to  measure  the  perceived  value  of  the  curriculum.  For  the  purposes 
of  this  study,  a  protocol  analysis  was  conducted  to  pilot  test  the  survey.  The  final 
product  of  this  thesis  is  a  smooth  copy  of  the  two  surveys.  The  surveys  can  then 
be  used  by  the  resource  sponsors  or  in  a  follow-on  study  to  capture  the  value  of 
the  MSA  curriculum  to  the  Navy  and  the  individual.  This  retrospective  look  could 
prove  very  useful  in  revising  the  curriculum  and  meeting  changing  needs  of  the 
Navy. 
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The  graduate  survey  will  focus  primarily  on  two  of  the  four  levels  of 
Kirkpatrick’s  training  program  evaluation:  reaction  and  learning.  (Kirkpatrick 
1998)  Graduates  are  asked  questions  about  how  the  curriculum  changed  them, 
either  through  learned  skills  or  attitudinal  changes. 

When  management  discusses  the  effectiveness  of  training  programs,  they 
usually  use  Kirkpatrick’s  behavioral  or  results  perspectives.  The  survey  for  the 
supervisors  will  primarily  focus  on  behavioral  criteria  because  evaluation  of  the 
economic  Return  on  Investment  (ROI)  metric,  which  involves  estimating 
economic  costs  and  benefits,  is  outside  the  scope  of  this  study.  Supervisors  are 
asked  questions  to  determine  how  prepared  the  MSA  graduates  are  compared  to 
other  graduates  and  whether  the  MSA  graduates  meet  their  expectations. 
Questions  were  designed  to  measure  differences  in  performance  between 
graduates  and  non-MSA  graduates. 

E.  RESEARCH  QUESTIONS 

The  direct  research  questions  that  will  be  addressed  in  the  scope  of  this 
thesis  are:  (1)  what  kinds  of  metrics  have  been,  or  could  be,  used  for  evaluating 
training  and  education?;  and  (2)  what  are  the  aspects  of  the  MSA  curriculum  that 
should  be  assessed  in  a  survey  to  determine  the  value  to  the  Navy  of  the  MSA 
curriculum? 

F.  ORGANIZATION  OF  THE  THESIS 

This  thesis  is  organized  into  four  chapters.  Chapter  II  briefly  reviews 
methods  for  measuring  the  return  on  human  capital  investments,  training 
evaluation  models,  and  discusses  prior  studies  on  graduate  education.  This 
chapter  also  discusses  traditional  Return  on  Investment  cost  analysis  of  training, 
Knowledge  Value  Added,  and  survey  writing  techniques.  Chapter  III  describes  in 
detail  the  Manpower  Systems  Analysis  curriculum,  the  target  population,  and  the 
measurement  tools  that  are  created--the  two  surveys  or  final  output  of  the  thesis 
research.  This  chapter  also  discusses  the  methods  for  evaluating  the  surveys. 
Chapter  IV  summarizes  the  results  of  the  research  and  makes  recommendations 
for  future  studies. 


11 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


12 


II.  LITERATURE  REVIEW 


A.  MEASURING  THE  RETURN  ON  HUMAN  CAPITAL  INVESTMENTS 

Many  different  approaches  have  been  used  to  measure  investments  in 
human  capital.  However,  “there  are  no  generally  accepted  accounting 
procedures  for  valuation  of  human  assets — employees.”  (Carnevale  and  Schulz 
1990,  P.230)  Some  organizations  have  attempted  an  employee  valuation, 
distributing  costs  of  recruitment  and  training  across  the  expected  length  of 
employment.  Other  organizations  use  replacement  costs  of  an  employee.  Yet 
another  way  to  evaluate  human  capital  investments  is  to  estimate  the  monetary 
value  of  changes  in  employee  behavior  due  to  training.  (Carnevale  and  Schulz 
1990,  pp. 230-231)  This  chapter  demonstrates  some  of  the  different  models  that 
can  be  used  to  evaluate  training  and  education  and  discusses  different  methods 
of  measuring  the  return  on  human  capital  investments.  In  addition,  some 
Department  of  Defense  and  corporate  examples  of  training  and  education 
evaluations  are  included. 

1.  Training  Evaluation  Models 

a.  Kirkpatrick’s  Four  Levels  of  Evaluation 
It  is  not  enough  to  simply  conduct  training  in  an  organization,  but 
that  program  must  also  be  evaluated  to  determine  its  effectiveness.  (Kirkpatrick 
1998b,  p.1)  Donald  L.  Kirkpatrick  identifies  three  reasons  for  why  we  should 
evaluate  training:  “to  justify  the  existence  of  the  training  department... to  decide 
whether  to  continue  or  discontinue  training  programs  and  to  gain  information  on 
how  to  improve  future  training  programs.”  (Kirkpatrick  1998b,  p.1 6)  In  order  to 
conduct  this  evaluation,  Kirkpatrick  establishes  four  levels  training  programs 
should  be  evaluated  on:  reaction,  learning,  behavior,  and  results. 

The  level-one  evaluation,  the  reaction  stage,  measures  how  the 
trainees  feel  about  a  program;  this  is  a  “measure  of  customer  satisfaction.” 
(Kirkpatrick  1998b,  p.1 9)  This  evaluation  is  often  conducted  using 

questionnaires  given  to  trainees  after  the  training  is  complete.  Many  evaluators 
call  these  questionnaires,  “happiness  sheets”  or  “smile  sheets.”  (Kirkpatrick 
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1998b,  pp.25,  28)  Often  people  view  these  surveys  as  worthless;  however,  they 
do  provide  valuable  information.  Many  times,  positive  feedback  is  all  that  is 
required  to  continue  a  program.  Kirkpatrick  also  states  that  if  trainees  do  not 
react  “favorably”  to  a  training  program,  they  usually  are  not  motivated  to  learn. 
Kirkpatrick  states  four  reasons  why  it  is  important  to  measure  reactions:  (1)  the 
feedback  can  be  used  to  evaluate  the  current  program  and  improve  future 
programs;  (2)  involves  trainees  in  improving  the  program,  tells  them  that  their 
feedback  is  important;  (3)  gives  quantitative  data;  and  (4)  results  can  be  used  to 
establish  performance  standards  for  future  training.  (Kirkpatrick  1998b,  p.25)  To 
understand  trainees’  reactions,  Kirkpatrick  suggests  using  open-ended  questions 
in  addition  to  questions  that  can  be  tabulated.  Without  these  open-ended 
questions,  it  is  difficult  to  understand  reasons  for  the  reactions  and  to  identify 
areas  for  program  improvement. 

After  measuring  reactions  to  the  training  programs,  Kirkpatrick  says 
that  “learning”  must  be  evaluated.  This  level  measures  the  “extent  to  which 
participants  change  attitudes,  improve  knowledge,  and/or  increase  skill  as  a 
result  of  attending  the  program.”  (Kirkpatrick  1998b,  p.20)  This  step  is  vital 
because  if  learning  does  not  take  place,  behavior  cannot  change  and  no  results 
to  the  company  can  be  realized.  Kirkpatrick  recommends  using  a  before-and- 
after  training  measure  of  job  performance  to  quantify  changes  in  knowledge, 
skills,  attitudes,  and  behavior  after  training. 

Although  this  is  ideal,  it  is  not  entirely  feasible  in  the  Navy  in 
regards  to  the  MSA  curriculum.  The  way  the  Navy’s  billet  system  works,  a 
person  would  not  return  to  the  same  job  after  completion  of  graduate  education. 
Therefore,  a  before-and-after  performance  measurement  on  the  part  of  the 
supervisor  would  not  be  possible.  Even  if  FITREPS  were  used  as  a 
measurement  tool,  they  would  not  accurately  portray  changed  behavior.  The 
Navy’s  evaluation  system  is  based  on  the  opinions  of  supervisors,  which  vary 
from  person  to  person.  Because  the  supervisors  do  not  stay  the  same,  the 
FITREP  would  not  work  as  a  before-and-after  form  of  evaluation.  Where  before- 
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and-after  measurement  will  be  utilized  is  through  self-evaluations.  Each 
graduate  will  be  asked  to  rate  his  or  her  own  behavior  changes  as  a  result  of  the 
MSA  curriculum.  The  graduate  will  be  asked  to  assess  how  specific  behaviors 
changed  compared  to  that  prior  to  entering  graduate  education. 

When  measuring  learning,  it  is  recommended  to  use  a  control 
group  if  it  is  feasible.  The  control  group  is  a  group  who  did  not  receive  the 
training  but  with  similar  characteristics  as  the  group  that  did  receive  the  training. 
A  comparison  is  then  conducted  between  the  behavior  of  those  trained  (the 
experimental  group)  with  the  untrained  group.  (Kirkpatrick  1971,  p.  96)  This 
control  group  is  used  as  a  comparison  in  an  attempt  to  eliminate  other 
contributing  factors  when  measuring  change.  This  can  be  done  a  bit  more  easily 
in  the  Navy  than  conducting  a  before-and-after  evaluation.  In  the  survey  given  to 
the  graduates  supervisors,  there  are  a  series  of  questions  asking  them  to 
compare  their  graduates  to  other  non-graduate  subordinates.  In  essence,  these 
non-graduates  are  the  control  group  (not  receiving  the  training)  and  the 
graduates  are  the  experimental  group. 

After  “learning”  has  been  measured,  an  evaluator  can  determine  if 
the  training  resulted  in  any  behavior  changes.  Although  Kirkpatrick  does  suggest 
that  the  evaluation  levels  be  conducted  in  order,  Kirkpatrick  states  that 
conducting  one  level  of  evaluation  does  not  eliminate  the  need  for  another.  For 
example,  many  people  believe  that  if  they  conduct  a  level-three  evaluation,  they 
need  not  conduct  a  level-two,  or  learning  evaluation.  Kirkpatrick  strongly 
recommends  against  this.  He  argues  that  there  are  several  cases  where 
learning  has  taken  place  but  behavior  does  not  change.  If  learning  were  not 
measured  then  the  evaluator  would  assume  that  the  training  was  not  effective. 
Behavior  changes  might  not  occur  if:  (1)  the  person  does  not  want  to  change;  (2) 
the  person  does  not  know  what  or  how  to  change;  (3)  the  corporate  climate  is  not 
conducive  to  change;  or  (4)  the  person  is  given  no  reward  for  change  (either 
intrinsic  or  extrinsic).  (Kirkpatrick  1998b,  pp.2 1 ,  37)  In  order  to  evaluate 
behavior,  Kirkpatrick  again  suggests  using  a  control  group  and  testing  before  and 
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after  the  training.  He  also  suggests  that  in  addition  to  questioning  trainees, 
evaluators  could  survey  immediate  supervisors,  subordinates,  and  others  who 
might  observe  changes  in  behavior.  Although  many  people  believe  a  trainee’s 
supervisor  would  be  the  best  to  evaluate  a  trainee’s  behavior  changes,  in  some 
cases,  Kirkpatrick  states  that,  “the  immediate  supervisor  may  be  the  person  least 
qualified  to  evaluate  the  trainee.”  (Kirkpatrick  1998b,  p.51)  This  train  of  thought 
coincides  with  the  new  performance  evaluation  trend  of  360-degree  evaluations, 
where  a  person’s  performance  is  measured  by  everyone  who  comes  in  contact 
with  that  individual  (supervisors,  peers,  subordinates,  customers).  When 
evaluating  this  change,  it  is  also  important  to  give  time  for  the  training  to  take 
effect.  It  depends  on  the  type  of  training  that  is  given  as  to  when  the  trainees 
should  be  evaluated  for  behavioral  changes.  For  example,  he  states  that  for 
some  training,  a  good  “rule  of  thumb”  for  evaluation  is  after  two  or  three  months 
and  for  others,  after  six  months  is  more  practical.  Kirkpatrick  also  suggests 
repeating  the  evaluation  at  different  time  intervals. 

Some  trainees  may  change  their  behavior  as  soon  as  they  return  to 
their  job.  Others  wait  six  months  or  a  year  or  never  change.  And 
those  who  change  immediately  may  revert  to  the  old  behavior  after 
trying  out  the  new  behavior  for  a  period  of  time.  Therefore,  it  is 
important  to  repeat  the  evaluation  at  an  appropriate  time. 
(Kirkpatrick  1998b,  p.55) 

In  order  to  measure  this  desired  change  in  behavior,  Kirkpatrick  suggests  using 
surveys  or  interviews,  or  a  combination  of  the  two.  He  identifies  change  in 
behavior  as  “a  means  to  an  end.”  (Kirkpatrick  1998b,  p.56)  The  “end”  is  the 
desired  changes  in  an  organization,  which  can  only  be  achieved  through 
changed  behavior. 

These  coveted  changes  in  an  organization  are  typically  the  reason 
for  the  training  in  the  first  place.  This  final  level  of  evaluation,  the  results  stage,  is 
often  the  most  difficult,  but  most  requested  evaluation.  Top  management  often 
requests  the  results  of  a  training  program  based  on  balance  sheet  figures. 
Sometimes  results  of  training  programs  can  be  easily  measured  such  as 
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increased  production,  sales  or  profits;  and  reduced  turnover,  costs  or  failure 
rates.  Kirkpatrick  also  notes  that  often  results  can  only  be  seen  on  a  long-term 
basis  such  as  improved  morale  or  quality  of  work  life.  Although  it  is  difficult  to 
quantify  these  improvements,  it  is  implied  that  these  changes  will  cause  the 
desired  tangible  results  in  the  future.  There  are  a  series  of  guidelines  that  are 
established  when  evaluating  results.  These  guidelines  are  similar  to  those  for 
evaluating  change.  It  is  recommended  to  use  a  control  group  and  measure  both 
before  and  after  the  training  is  conducted.  Also,  as  with  the  behavior  level,  it  is 
important  to  allow  time  for  the  results  to  occur  and  to  re-measure  at  different  time 
intervals.  Kirkpatrick  also  warns  the  evaluator  to  weigh  the  cost  of  evaluation 
against  the  benefits.  If  it  is  too  costly  to  quantify  the  results  of  a  program,  and  the 
first  three  levels  are  sufficient,  then  the  benefit  of  conducting  an  evaluation  of 
results  does  not  outweigh  the  costs.  If  the  program  is  going  to  be  conducted 
often,  then  more  costly  evaluation  might  be  justified.  In  addition,  the  evaluator 
must  determine  what  level  of  certainty  is  required.  If  management  would  be 
satisfied  with  evidence  of  positive  results  of  the  training,  then  extensive,  costly 
evaluation  is  not  needed.  Kirkpatrick  further  warns  that  proof  is  often  not  feasible 
and  one  must  settle  for  evidence  of  improvement. 

b.  Martin  Marietta’s  Five  Step  Model 

A  common  complaint  about  Kirkpatrick’s  four-level  model  is  that 
evaluation  is  often  looked  at  only  from  the  perspective  of  how  good  the  training 
is,  not  on  job  performance  improvements.  According  to  Marshall  and  Schriver 
(1994,  p.  1 27)  from  Martin  Marietta,  this  downfall  leads  to  evaluation  based  on 
simply  the  reaction  of  the  trainees  to  the  training.  The  lip  service  that  many 
companies  give  to  conducting  training  as  a  requirement,  rather  than  a  way  to 
improve  on  skills  that  are  lacking,  leads  companies  to  evaluate  training  purely  in 
the  first  two  levels,  reactions  and  learning.  (Marshall  and  Schriver  1994,  p.  127) 

Often  evaluators  oversimplify  Kirkpatrick’s  model,  by  evaluating 
only  knowledge  based  on  written  test  results  and  failing  to  measure  the  trainee’s 
skills  gained.  Martin  Marietta  Energy  Systems  developed  a  five-step  model  to 
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better  evaluate  both  knowledge  and  skills.  This  model  separates  the  evaluations 
into  two  sections,  formative  and  summative.  The  first  three  sections  are  the 
formative  parts,  measuring  attitudes  or  feelings  (similar  to  Kirkpatrick’s  reaction 
stage),  knowledge  and  skills  (both  part  of  Kirkpatrick’s  learning  stage).  The 
summative  part  of  the  model  consists  of  measuring  skills  transfer,  or  behavior 
modification  (Kirkpatrick’s  Behavior  stage)  and  Organizational  impact 
(Kirkpatrick’s  results  stage).  (Marshall  and  Schriver  1994,  p.  129)  Martin 
Marietta’s  Model  for  Evaluating  Knowledge  and  Skills  can  be  seen  below  in 
Figure  2. 
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Figure  2:  Model  for  Evaluating  Knowledge  and  Skills  (From:  Marshall  and 
Schriver  1994). 


c.  Training  Impact  Tree 

Paul  Bernthal,  a  research  consultant  from  Development 

Dimensions  International,  argues  that  Kirkpatrick’s  four-level  model  is  too 
simplistic  and  has  some  inherent  pitfalls.  “...It  has  also  limited  our  thinking 

regarding  evaluation  and  possibly  hindered  our  ability  to  conduct  meaningful 

evaluations.”  (Bernthal  1995,  p.50)  Bernthal  (1995)  points  out  that  there  are  a 
series  of  false  assumptions  about  four-level  evaluations.  Many  people  believe 
that  evaluations  are  conclusive  and  reliable;  however,  he  says  often  the  results 
could  not  be  duplicated.  The  second  assumption  is  that  “evaluation  equals 
effectiveness.”  (Bernthal  1995,  pp.50)  He  states  that  although  the  two  are 
associated,  they  are  not  that  easy  to  measure.  To  fully  capture  the  effectiveness 
of  a  training  program,  the  evaluator  must  look  at  different  organizational, 
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individual  and  training-related  factors.  Another  big  assumption  companies  often 
make  is  that  the  trainers  are  accountable  for  effectiveness  of  the  training. 
Bernthal  (1995)  states  that  trainers  often  do  not  possess  the  skills,  time  and 
resources  necessary  for  thorough  evaluations,  yet  are  still  held  responsible. 
Possibly  the  biggest  argument  Bernthal  has  against  Kirkpatrick’s  four-level  model 
is  based  on  how  many  people  interpret  the  model.  Often,  Kirkpatrick’s  fourth 
level  is  described  as  the  highest  or  best  type  of  evaluation.  Bernthal  argues  that 
each  level  has  equal  value  in  different  situations. 

Bernthal  suggests  a  series  of  additions  to  the  four-level  evaluation 
model.  He  states  that  the  evaluator  needs  to  understand  where  the  training  fits 
in  the  organization,  determine  the  cause-and -effect  between  training  and 
outcomes,  and  choose  appropriate  evaluation  designs  and  measurements  for 
different  steps  in  the  evaluation  process.  Also,  the  evaluator  must  remain 
realistic  about  the  time  and  resources  that  can  be  allotted  to  evaluation  and 
should  create  an  evaluation  program  for  long-range  use.  (Bernthal  1995,  pp.52- 
53)  Bernthal  lists  a  series  of  seven  steps  to  use  in  conjunction  with  Kirkpatrick’s 
four-level  model:  (1)  Identify  the  organizations’  values  and  practices;  (2)  Identify 
the  desired  knowledge,  skills,  and  attitudes;  (3)  Determine  the  purpose  and 
scope  of  the  evaluation;  (4)  Identify  the  data  sources;  (5)  Select  the  best  method 
for  data  collection,  taking  into  account  how  often  the  evaluation  will  occur  and 
how  many  people  will  be  evaluated;  (6)  Choose  the  best  measurement  approach; 
and  (7)  Gather  and  Inventory  your  resources.  (Bernthal  1995,  pp. 52-53) 

To  improve  on  some  of  the  pitfalls  Bernthal  finds  with  Kirkpatrick’s 
four-level  model,  he  recommends  several  areas  that  should  also  be  evaluated: 
“the  quality,  delivery,  or  retention  of  the  training;  how  well  the  training  cut 
deficiencies  in  a  particular  work  group;  the  usefulness  of  parallel  training  for 
managers  and  their  staffs;  and  variables  in  the  work  environment  that  discourage 
or  facilitate  the  effect  of  training.”  (Bernthal  1995,  p.  52).  He  suggests  using  a 
“training-impact  tree”  to  look  at  some  of  these  factors  when  developing  the  long- 
range  plan.  The  tree  is  a  planning  tool  used  to  identify  any  factors  that  could 
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affect  training  and  evaluation  within  an  organization.  The  Training  Impact  tree  is 
essentially  identifying  the  organization’s  values,  barriers  to  training,  and  factors 
that  will  aide  in  training.  The  final  portion  of  the  Training-Impact  Tree  is  listing  the 
knowledge,  skills  and  attitudes  the  training  is  designed  to  teach.  An  example  of 
the  Training-Impact  Tee  can  be  seen  below  in  Figure  3. 


A  Training-impact  Tree 

Here's  an  example  of  a  training  impact  tree.  It  can  help  Identify  the  variables  in  an  organization 
that  might  affect  training  outcomes. 


Barriers  to  Training 


»  Organizational  values  aren't  dearly 
communicated. 

I  Senior  managers  send  mixed 
messages. 

*  Achieving  goals  requires  competitive 
behaviors  from  employees. 

►  Managers  aren't  held  accountable  for 
their  teams'  success. 

►  Team  members  work  on  different  floors 
or  in  different  buildings. 


Organizational  Values 

Example:  teamworK 

Supporting  practices  include  the  following: 

►  recognizing  team  efforts 

I  participating  in  cross-functional  teams 

►  linking  individual  goals  to  group  goals 

►  dialing  responsibility  and  accountability 
for  learn  outputs 

*  exchanging  information  openly  and 
frequently 

I  establishing  partnerships  with  other 
departments  and  teams. 


Factors  in  Facilitating  Training 


»  The  organization  has  a  good  relation¬ 
ship  with  its  unions. 

>  The  organization  has  a  stable  profit 
margin. 

»  1  he  organization's  values  are  logical. 

»  The  job  descriptions  are  dear. 

►  The  selection  system  includes  effective 
teamwork  as  a  criterion. 

»  Managers  serve  as  positive  role  models. 
*  The  performance-management  system 
rewards  team  accomplishments. 


Cognitions  (knowledgo  and  thoughts) 

I  the  role  of  team  leaders 
I  the  effect  of  Individual  members  on  a 
group  as  a  whole 
I  the  meaning  of  empowermem 
»  the  way  teams  and  groups  work  together 
I  the  importance  of  sharing  activities  and 
tasks  amsng  group  members 
I  the  importance  of  diversity. 


Training 

Skills 

I  expresses  thoughts  clearly 
k  listens  and  responds  with  empathy 
k  seeks  input  from  ethers 
*  offers  help  when  others  need  It 
a  makes  sure  all  team  members  are  heard 
and  understood 

a  shares  ideas  without  being  askod 
a  regularly  recognizes  individual  efforts  In 
support  of  team  goals. 


Feelings 

a  trie  motivation  to  grow  and  develop 
a  a  sense  of  team  spirit 
a  self-efficacy 

a  a  concern  for  group  cohesion. 


Figure  3:  Training  Impact  Tree  (From:  Bernthal  1995) 


d.  Input,  Process  Output 

Increasingly,  organizations  are  moving  away  from  using  only  the 
standard  four-level  model  proposed  by  Kirkpatrick.  According  to  David  Bushnell 
(1990),  a  director  at  the  Human  Resources  Research  Organization,  many 
industry  leaders  are  now  developing  an  input-process-output  (IPO)  method  for 
evaluating  training.  This  model  is  used  to  see  if  training  programs  are  achieving 
their  intended  purposes  and  as  a  way  to  determine  whether  students  acquire  the 
needed  skills  and  knowledge.  The  IPO  Evaluation  Model  is  based  on  the  idea 
that  every  training  program  has  an  input,  a  process  and  an  output  and  should  be 
evaluated  at  each  of  these  points.  (Bushnell  1990,  p.40)  Bushnell  describes  the 
factors  that  could  contribute  to  the  training  effectiveness  as  the  input  factors,  or 
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the  system  performance  indicators.  These  factors  include  trainee  qualifications, 
instructor  abilities,  the  quality  of  instructional  materials,  availability  of  training 
facilities  and  money.  In  the  process  stage,  the  planning  and  implementation  of 
the  training  takes  place.  The  output  evaluation  stage  captures  the  “short  term 
benefits  of  the  training.”  (Bushnell  1990,  p.40)  This  stage  includes  trainee 
reactions  and  the  harder  to  measure  knowledge  and  skills  gained  and  improved 
job  performance.  The  outcomes  of  training  are  generally  longer  term,  including 
company  profits,  customer  satisfaction,  and  productivity.  Bushnell  states  that 
feedback  loops  must  exist  through  the  training  process  at  each  critical  stage  in 
order  to  make  the  training  system  “somewhat  self-correcting.”  (Bushnell  1990,  p. 
40)  The  IPO  model  Bushnell  introduces  can  be  seen  in  Figure  4,  below. 


Figure  4:  IPO  Training  Approach  (From:  Bushnell  1990,  p.40) 

2.  Return  on  Investment  Cost  Analysis  of  Training 

Many  managers  want  a  more  exact  measurement  of  value  of  training  and 
education.  This  frame  of  mind  is  displayed  in  the  words  of  one  Fortune  100 
financial  Vice  President  to  his  Director  of  Human  Resource  Development.  “I 
invest  in  training  the  way  I  invest  in  a  machine  tool.  If  you  can’t  show  me  an  ROI 
equal  to  this  firm’s  cost  of  capital,  I’m  not  buying — and  your  budget  is  going  to  be 
cut.”  (Spencer  1986  p.1 )  Measuring  the  return  on  investment  for  training  is  a  way 
to  show  the  value  of  an  organization’s  training  directly  as  it  relates  to  the 
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company’s  bottom  line.  There  are  four  reasons  for  cost-benefit  evaluation: 
improve  practice;  survival;  credibility;  professional  development;  and  satisfaction. 
(Spencer  1986,  p.2)  When  cost  benefit  analysis  is  conducted,  human  resource 
personnel  and  activities  productivity  can  increase  by  30  to  50  percent.  (Spencer 
1986,  p.2)  Even  though  there  are  such  positive  reasons  for  conducting  return  on 
investment  cost  benefit  analysis,  these  studies  are  not  often  done... ’’most  people 
believe  these  data  are  hard,  expensive,  and  take  a  long  time  to  get.”  While  it  is 
true  that  these  computations  are  often  more  time  consuming,  they  can  be  a  great 
value  to  the  organization,  “more  valuable  measures  may  require  more  resources: 
you  get  what  you  pay  for.”  (Spencer  1986,  p.  1 8) 

Jack  Phillips  also  discusses  a  “Chain  of  effect”  when  it  comes  to 
evaluating  training.  He  uses  a  Five-Level  model  adapted  from  Kirkpatrick’s  four- 
level  model.  The  additional  element  includes  the  ROI  evaluation  step  as  the  fifth 
level.  (Phillips  1996b)  This  model  can  be  seen  in  Figure  5  below. 


Here's  a  slightly  modified  version  of  KirkpatriLk's  fou  -level  evaluation 
model,  adapted  to  include  measuring  for  return  on  investment. 

Laval 

Questions 

1.  reaccon  and  planned  action 

What  are  participants'  reactions  to 
the  training?  What  do  they  plan  to  do 
with  the  material? 

2.  learning 

What  skills,  knowledge,  or  attitudes 
have  changed?  By  how  much? 

3.  co-che-job  application 

Did  participants  apply  on  the  job  what 
they  learned  in  training? 

4.  business  results 

Did  the  on-the-job  applications  pro¬ 
duce  measurable  results? 

5.  return  on  Investment 

Did  the  monetary  value  of  the  results 
exceed  the  cost  of  training? 

Figure  5:  Five-Level  ROI  Evaluation  (From:  Phillips  1996) 

It  is  difficult  to  measure  the  effect  of  training  on  performance  because 

often  other  variables  also  contribute  to  changes  in  results.  The  best  way  to 
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account  for  other  variables  is  to  use  a  control  group  of  people  who  did  not 
receive  the  training.  If  this  cannot  be  done,  another  option  is  to  have  the 
participants  in  the  training  estimate  how  much  of  their  improvement  is  due  to  the 
training  they  received.  The  subjects  should  also  be  asked  what  their  estimation 
is  based  on  and  how  certain  they  are  in  the  estimation.  (Phillips  1996b,  p.217- 
218) 

To  determine  the  Return  on  Investment,  the  costs  of  a  training  program 
are  subtracted  from  the  monetary  benefits  of  the  program,  which  yields  the  net 
benefits  of  the  training.  The  net  benefits  are  then  divided  by  the  costs.  (Phillips 
1996,  p.210)  There  are  five  steps  to  calculating  the  return  on  investment  of  a 
program.  First,  the  level  four  results  should  be  determined.  Then,  the  evaluator 
needs  to  try  to  isolate  the  factors  that  contributed  to  the  training  from  other 
external  factors  that  might  have  affected  the  results.  This  is  most  easily  done  if  a 
control  group  is  used.  The  evaluator  then  needs  to  try  to  measure  the  results  as 
related  to  monetary  benefits  to  the  company.  The  evaluator  needs  to  then 
determine  the  cost  of  the  training,  including  lost  opportunity  costs  of  the  trainees 
and  trainers.  The  final  step  of  calculating  ROI  is  to  compare  the  costs  of  the 
training  with  the  benefits  of  the  program.  (Phillips  1996,  p.220c)  A  framework  for 
developing  an  ROI  model  can  be  seen  in  Figure  6. 
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Figure  6:  A  Model  for  Calculating  ROI  (From:  Phillips  1996) 


Phillips  describes  the  results  that  are  measured  to  determine  ROI  as  hard 
and  soft  data.  Hard  data  are  usually  easier  to  measure  in  an  organization. 
These  include  standard  measures  of  performance  in  an  organization  such  as: 
increased  production  output;  reduced  material  waste  or  rework;  money  saved; 
and  increased  sales.  These  factors  are  usually  easily  converted  to  bottom  line 
financials  and  are  objective.  Soft  data  on  the  other  hand  are  less  easy  to 
measure  and  difficult  to  quantify.  Soft  data  often  has  to  do  with  employee 
behavior  and  attitudes  such  as  employee  absenteeism  or  tardiness  and  job 
satisfaction.  Management  often  puts  little  credibility  in  using  soft  data  as  a 
performance  measure.  (Phillips  1996,  p  20) 

3.  Knowledge  Value  Added  (KVA) 

Return  on  Investment  calculations  are  essential  in  a  production  company 
but  when  the  asset  to  an  organization  is  intangible,  measurement  proves  to  be 
more  difficult.  Knowledge  is  a  fundamental  asset  of  the  Information  Age  as  we 
move  from  a  nation  of  production  to  service.  (Housel  and  Bell  2001,  p.40)  With 
this  transformation,  knowledge  measurement  is  essential.  Housel  and  Bell 
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describe  knowledge  as  a  “fundamental  unit  of  measurement,”  which  needs  to  be 
tracked  through  each  operation.  Knowledge  value  added  (KVA)  is  an  approach 
designed  to  estimate  the  value  of  knowledge  in  an  organization.  The  KVA 
approach  is  conducted  using  ratio  of  the  knowledge  needed  for  a  required  output 
in  an  organization.  Housel  and  Bell  describe  the  KVA  ratio  as  follows: 

This  is  accomplished  through  a  return  ratio  with  the  numerator  of 
the  ratio  being  the  percentage  of  the  revenue  or  sales  dollar 
allocated  to  the  amount  of  knowledge  required  to  obtain  the  outputs 
of  a  given  process  in  proportion  to  the  total  amount  of  knowledge 
required  to  generate  the  corporation’s  salable  outputs.  The 
denominator  of  the  ratio  is  the  cost  to  execute  the  process 
knowledge.  (Housel  and  Bell  2001,  p.40) 

Subject-matter  experts  within  the  organization  are  interviewed  and  all 
processes  within  the  organization  are  observed.  It  is  determined  how  much  time 
is  spent  learning  each  process,  the  monetary  value  each  process  adds,  and  the 
cost  spent  teaching  the  process  and  how  long  the  process  takes.  Including  all 
these  computations  allows  for  knowledge  to  be  measured.  KVA  provides  a 
standard  output  in  terms  of  units  of  knowledge  for  each  process.  A  great 
advantage  of  KVA  is  that  knowledge  is  expressed  in  a  common  unit  across  all 
jobs  and  processes. 

4.  Economic  Value  Added  (EVA) 

Author,  and  president  of  Imperial  Corporate  Training  and  Development, 
Edward  Gordon  (1996)  discusses  that  there  are  several  ways  to  measure  value 
of  educational  programs,  a  way  to  measure  how  much  a  training  program  is 
worth,  calling  it  economic  value  added  (EVA).  One  type  of  EVA  Gordon 
mentions  is  “Utility  analysis.”  (Gordon  1996,  p.69)  He  describes  utility  analysis 
as  “a  hard-data  method  for  determining  return  on  investment  (ROI)  for  training  by 
calculating  the  value  of  an  intervention  (i.e.,  a  training  program)  minus  its  costs.” 
This  economic  gain,  or  the  added  value,  is  computed  by  taking  the  outcome  of 
the  training,  multiplied  by  the  dollar  value  of  the  training  effect.  Gordon’s 
equations  can  be  used  to  conduct  a  cost-benefit  analysis  of  the  training  program. 
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This  computation  can  be  seen  in  the  utility  analysis  equation  below.  (Gordon 
1996,  p.69)3 

F  =  N[(E*M)-C\ 

F  =  financial  utility 

N  =  number  of  people  trained 

E  =  effect  of  the  training  on  the  business 

Another  form  of  economic  value  added  is  the  Performance  value.  This 
method  attempts  to  quantify  the  monetary  value  of  problems  that  are  to  be 
addressed  by  the  training.4  Total  Quality  Management  (TQM)  and  Six  Sigma 
approaches  are  examples  of  ways  performance  value  is  being  used.  Companies 
trained  employees  to  improve  quality  in  manufacturing,  which,  in  turn,  increased 
company  profits.  “Poor  quality  in  products  or  services  may  often  cost  tens  of 
percents  of  economic  value  added.”  (Gordon  1996,  p.69) 

The  value  of  education  to  a  manufacturing  firm  can  be  realized  more 
easily  when  profits  or  quantity  of  production  is  measurable.  One  of  the  biggest 
challenges  facing  companies  today  is  when  this  economic  value  added  cannot 
be  traced  directly  to  a  balance  sheet  or  bottom  line  company  profits. 

There  are  several  additional  computations  that  will  yield  the  monetary 
value  of  human  capital.  In  his  book,  The  ROI  of  Human  Capital,  Jac  Fitz-enz 

(2000)  identifies  a  series  of  equations  that  can  be  used  in  addition  to  the  utility 

analysis  equation  above.  Fitz-enz  (2000,  p.32)  discusses  calculating  the  “HEVA” 
or  Human  Economic  Value  Added,  which  is  simply  the  cost  of  capital  subtracted 
from  the  net  operating  profit  after  taxes  divided  by  the  number  of  full  time 

employees.  This  can  be  seen  in  Equation  1  listed  below.  Another  formula  that 

can  be  used  to  measure  profitability  for  the  average  employee  is  “HCVA”  or 

3  For  a  more  detailed  explanation  of  the  utility  analysis  approach,  refer  to  Wayne  F.  Cascio’s 
1989  book,  Training  and  Development  in  Organizations,  and  Michael  Godkewitsch’s  1987  book, 
Training. 

4  For  more  information  on  Performance  Value,  see  Forecasting  Financial  Benefits  of  Human 
Resource  Development,  written  by  Richard  A.  Swanson  and  Deane  B.  Gradous  (1988). 

26 


C  =  cost  of  the  training  per  person 
M  =  montetary  value  of  the  training  effect 


Human  Capital  Value  Added,  which  is  expenses  (not  including  pay  and  benefits) 
subtracted  from  revenue,  all  divided  by  the  number  of  full  time  employees.  (Fitz- 
enz  2000,  p.35)  This  can  be  seen  below  in  Equation  2.  Fitz-enz  (2000,  p.36) 
also  suggests  using  “HCROI”  or  Human  Capital  Return  on  Investment 
calculations  to  look  at  the  return  on  investment  in  terms  of  profits  for  the  amount 
of  money  spent  on  employee  pay  and  benefits.  This  is  calculated  similarly  to 
HCVA,  but  the  numerator  is  divided  by  the  amount  of  pay  and  benefits.  This  can 
be  seen  below  in  Equation  3. 

^  Net  operating  profit  after  tax  -  Cost  of  capital 

Equation  1 :  HEVA  = - - - — - - - 

Number  of  Full  time  Employees  (FTEs) 

^  ,  ,  Revenue  -  (Expenses  -  Pay  and  Benefits) 

Equation  2:  HCVA  = - - — - - - - - 

Number  of  Full  time  Employees  (FTEs) 


Equation  3:  HCROI  = 


Revenue  -  (Expenses  -  Pay  and  Benefits) 
Pay  and  Benefits 


Unfortunately,  most  of  these  measurements  do  not  really  work  for  the 
Military  because  it  is  essentially  a  “Not-for  profit”  organization.  As  a  result,  we 
must  look  at  other  forms  of  evaluation. 


B.  TECHNIQUES  FOR  EVALUATION 

There  are  several  data  collection  tools  that  can  be  used  that  are  both 
qualitative  and  quantitative.  Some  quantitative  instruments  include:  performance 
records  and  tests;  standardized  questionnaires  and  survey  instruments;  and 
personnel  assessment  instruments.  Quantitative  data  is  generally  objective, 
relatively  easy  to  measure  and  assign  monetary  values,  uses  a  common 
measure  of  performance,  and  is  seen  as  credible  to  management.  (Carnevale 
and  Schulz  1990,  p.246)  Some  common  qualitative  instruments  are  interviews, 
observations,  focus  groups,  and  case  studies.  In  general,  qualitative  data  is 
usually  difficult  to  measure  and  standardize,  based  on  behavioral  factors  and  is 
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less  accepted  by  management.  Although  many  circumstances  call  for 
quantitative  analysis,  often  qualitative  data  provides  information  that  cannot  be 
attained  in  other  ways,  or  serves  as  a  valuable  supplement  to  the  quantitative 
instruments.  (Carnevale  and  Schulz  1990,  p.246) 

The  Senior  HR  Management  Specialist  for  the  Chief  Administrative  Office 
of  Los  Angeles  County  identifies  ten  different  instruments  or  tools  that  can  be 
used  to  evaluate  training.  (Marrelli  1993,  p.  58)  She  continues  to  identify  the 
strengths  and  weaknesses  of  each  evaluation  tool  and  warns  against  using  just 
one.  “Different  instruments  highlight  different  results  of  the  same  training 
program.  You  might  miss  some  of  the  results  by  using  only  one  instrument.” 
(Marrelli  1993,  p.  58)  The  ten  instruments  she  names  are:  Interviews, 
Questionnaires,  Group  Discussions  (focus  groups),  Critical  Incident  Reports, 
Work  Diaries,  Performance  Records,  Simulations/Role-Plays,  Observations, 
Written  Tests,  and  Performance  Tests. 

Getting  reports  from  supervisors  on  the  progress  of  trainees  is  an  ideal 
way  of  evaluating  training.  However,  these  reports  are  often  seen  as  too  time 
consuming  and  often  will  not  be  completed.  Also,  these  reports  are  often  seen 
as  subjective,  and  can  change  drastically  from  one  supervisor  to  another. 
(Garavaglia  1993,  p.75)  To  get  a  more  objective  measurement  of  the  training,  a 
survey  or  questionnaire  can  be  used.  Typically  supervisors  and  trainees  are 
questioned  but  trainees’  peers  could  also  be  given  the  survey  for  further 
evaluation.  This  would  be  one  step  towards  implementing  a  360-degree 
evaluation  of  the  training.  “Surveys  and  questionnaires  provide  different 
perspectives  from  all  of  the  trainees  and  supervisors,  without  costing  a  lot  in  time 
or  money.”  (Garavaglia  1993,  p.75) 

In  addition  to  performance  reports  and  surveys,  interviews  could  be 

conducted  of  the  trainees  and  supervisors.  These  interviews  could  be  face-to- 

face,  via  Video-Teleconferencing  or  over  the  telephone.  The  interviews  could  be 

conducted  in  groups  or  on  an  individual  basis.  (Garavaglia  1993,  p.75)  While 

interviews  might  work  on  a  small  scale  for  an  organization,  it  would  be  infeasible 
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in  the  Navy’s  Human  resource  community.  Although  these  surveys  are  intended 
for  all  Navy  Manpower  Systems  Analysis  graduates  and  their  supervisors, 
currently,  the  Navy’s  Human  Resource  community  is  one  of  the  largest 
communities  with  MPT  related  billets.  The  HR  community  has  over  750  billets 
spread  out  in  14  different  regions  within  the  U.S.  and  10  different  regions  outside 
the  continental  U.S.  Even  only  looking  at  billets  with  the  3130-subspecialty,  this 
number  is  about  60  throughout  the  world.  (HR  website,  billet  locations.)  The  vast 
number  and  geographic  dispersion  alone  make  interviews  an  unrealistic  option. 

Although  it  is  ideal  to  have  several  different  evaluation  methods,  this 
thesis  focuses  primarily  on  the  development  of  two  questionnaires.  Given  this 
direction,  it  is  important  to  look  at  some  of  the  advantages  and  disadvantages  of 
questionnaires,  as  outlined  by  Marrelli  (1993).  Some  of  the  biggest  advantages 
of  surveys  or  questionnaires  are  that  they  are  generally  easy  to  interpret  (if 
constructed  correctly)  and  results  can  be  attained  faster  than  many  other  forms 
of  evaluation.  Respondents  often  answer  more  honestly  due  to  anonymity  of 
responses.  Also,  a  large,  geographically  dispersed  population  can  be  questioned 
at  little  to  no  cost.  These  many  advantages  to  using  surveys  have  led  them  to  be 
a  common  evaluation  instrument;  however,  several  disadvantages  need  to  be 
considered.  While  the  anonymity  of  the  survey  serves  as  an  advantage,  it  can 
also  be  a  disadvantage  because  the  quality  of  the  data  depends  on  the  subjects’ 
honest  answer  response.  The  respondent  may  also  not  remember  accurately  or 
have  a  different  perception  of  the  situation  being  evaluated.  There  is  also  no  way 
to  determine  if  the  reader  of  the  questionnaire  understands  what  each  question  is 
asking  and  if  the  answer  choices  (when  the  survey  is  multiple  choice)  accurately 
portrays  the  respondents’  feelings.  One  of  the  biggest  disadvantages  is  the  low 
rate  of  return  for  surveys.  (Marrelli  1993,  p.  60) 

C.  SURVEY  WRITING  TECHNIQUES 

There  are  many  different  books  written  on  survey  writing  techniques. 
Unfortunately,  all  that  surveys  have  in  common  is  that  they  are  structured 
questionnaires  asked  either  individually  or  in  group  interviews.  What  is  needed, 


29 


but  is  non-existent,  is  a  set  of  standardized  professional  guidelines  for  survey 
development.  (Smith  2003,  P.  21) 

1 .  Types  of  Survey  Questions 

Questionnaires  can  use  a  combination  of  the  following  different  types  of 
questions:  multiple  choice,  multiple  answer,  ranked  questions,  open-ended 
questions,  and  scaled  questions.  (Johnson  1993,  P.140)  Multiple  choice 
questions  consist  of  two  or  more  mutually  exclusive  answers.  Multiple  answer 
questions  allow  the  respondent  to  select  one  or  more  answer,  all  that  may  apply. 
Ranked  questions  ask  the  respondent  to  rank  a  list  of  items.  These  lists  are 
usually  in  order  of  preference  or  frequency  of  use.  An  open-ended  question  is  a 
question  with  no  right  answer.  The  questions  are  asked  without  a  selection  of 
answers  to  choose  from.  The  respondent  answers  the  question  in  his  or  her  own 
words.  Open-ended  questions  can  provide  a  wealth  of  information,  but  it  is  often 
hard  to  tabulate  the  results.  Scaled  questions  usually  measure  how  strongly  a 
person  feels  about  a  subject.  There  are  three  types  of  scales  that  are  commonly 
used:  Semantic-differential  scales;  Diagrammatic  scales;  and  Likert  scales. 
Semantic-differential  scales  are  based  on  a  bipolar  scale  (only  two  answers).  An 
example  of  a  semantic-differential  scale  would  be  “agree/disagree.”  A 
diagrammatic  scale  is  a  grid  or  diagram  “on  which  respondents  indicate  their 
position  with  respect  to  a  statement.  Words  or  numbers  are  usually  not  included 
on  the  grid,  with  the  scale  expressed  instead  by  an  abstract  category  or 
continuum — for  instance,  with  symbols  or  pictures.”  (Johnson  1993,  P.140)  A 
Likert  scale  asks  respondents  if  they  agree  or  disagree  with  a  statement.  Likert 
scales  are  commonly  given  values  from  1  to  5,  with  word  explanations  for  the 
values.  One  example  of  a  Likert  scale  would  be  1  “very  important,”  2  “important,” 
through  to  five  being  “Not  at  all  important.”  “The  five-point  Likert  scale  probably 
is  the  most  widely  used  rating  scale.”  (Paul  and  Bracken  1995,  p.  150)  The 
surveys  developed  for  this  thesis  have  a  combination  of  multiple  choice 
questions,  open  ended  questions,  and  scaled  questions  using  a  Likert  scale. 
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2.  Testing  Surveys 

As  with  survey  development,  there  is  no  agreed  upon  way  to  pretest 
questionnaires  and  there  is  very  little  guidance  on  the  correct  way  to  conduct 
such  tests.  (Bolton  1993,  P.2)  Pretesting  is  conducted  using  a  small  sample  of 
the  population,  what  is  also  called  a  pilot  study  or  pilot  test.  The  pilot  study  is 
used  to  determine  where  problems  might  exist  with  question  wording.  There  are 
several  different  ways  to  conduct  a  pretest;  however,  the  most  effective  are 
usually  conducted  in  one-on-one  interviews.  (Bolton  1993,  P.2)  The  different 
methods  of  pilot  testing  include  “concurrent  ‘think  aloud’  interviews, 
paraphrasing,  retrospective  ‘think  aloud’  interviews,  confidence  ratings,  vignettes, 
response  latency  measurements,  and  sorting  tasks.”  (Bolton  1993,  p.2  from 
Campanelli  et  al.  1989,  Jobe  and  Mingay  1990,  Royston  et  al.  1986,  Royston 
1987).  Although  each  of  these  methods  are  useful,  some  are  more  difficult  to 
conduct  than  others. 

a.  Automatic  Coding  and  Observational  Monitoring 

A  study  identifying  a  method  for  pretesting  questionnaires  based  on 
the  automatic  coding  of  verbal  and  nonverbal  cues  was  conducted  by  Ruth 
Bolton  of  GTE  Laboratories  Inc.  This  research  proposed  a  new  method  to 
identify  survey  questions  that  are  cognitively  difficult  for  the  respondent, 
measuring  their  reactions  to  a  questionnaire.  The  researchers  used  two  pilot 
groups  consisting  of  residential  telephone  company  customers  during  1989.  For 
each  pilot  testing,  face-to-face  interviews  were  conducted. 

In  1990,  taking  the  results  from  the  1989  pilot  tests,  GTE  conducted 
two  additional  pretests  to  evaluate  two  versions  of  a  survey  for  GTE’s  business 
customers.  Both  pretests  were  conducted  using  one-on-one  interviews  at  focus 
group  facilities  in  either  Dallas  or  Los  Angeles.  The  group  consisted  of  28 
interviews  and  the  second  group  consisted  of  30  interviews.  Each  of  the 
interviewers  received  training  on  how  to  conduct  the  interviews.  Each 
respondent  was  asked  to  read  the  question  out  loud  and  explain  what  they 
thought  about  the  questions. 
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In  both  sets  of  pilot  tests,  the  respondents’  reactions  (verbal  and 
nonverbal)  to  the  survey  were  measured  using  various  coding  categories.  The 
five  verbal  categories  that  were  used  were:  repeat,  forget,  confidence,  can’t  say, 
and  don’t  know.  The  four  non-verbal  categories  that  were  used  were:  questions, 
pauses,  broken  utterances  and  unintelligible  utterances.  (Bolton  1993,  P.4-5) 
Each  coding  category  was  then  used  to  evaluate  the  questions  based  on  the  four 
stages  of  macroprocesses  a  person  uses  to  react  to  an  attitude  question.  These 
four  stages  identified  by  Touraneau  are:  “comprehension  of  the  question,  the 
retrieval  of  relevant  beliefs  and  feelings  from  memory,  the  weighing  of 
information  to  form  a  judgment,  and  the  selection  of  an  appropriate  response 
alternative.”  (Bolton  1993,  p.4  from  Tourangeau  1987)  The  results  of  both  studies 
were  that  GTE  determined  the  best  indicator  of  a  person’s  comprehension  is  the 
non-verbal  cues  in  the  question  category  and  the  verbal  cues  in  the  repeat 
category,  meaning  the  person  either  looked  like  they  had  a  question  or  they 
asked  the  interviewer  to  repeat  or  clarify  the  question. 

Examining  the  retrieval  difficulties  of  respondents,  the  interviewers 
looked  for  the  verbal  cues  of  broken  utterances  and  the  nonverbal  cues  of  a 
pause  greater  than  three  seconds.  The  rationale  behind  the  broken  utterance 
category  is  that  if  people  change  their  thought  processes  in  mid-sentence,  the 
question  brings  two  answers  to  mind  and  thus  might  need  revising  or  should  be 
an  open-ended  question.  Bolton  found  that  when  evaluating  a  question  at  the 
judgment  stage  of  macroprocesses,  an  evaluator  should  look  at  the  verbal  cues 
of  confidence,  or  a  lack  of  confidence  in  the  responses.  The  nonverbal  cue 
would  be  an  unintelligible  utterance  such  as  “urn”  or  “er.”  To  determine  if  the 
response  to  a  question  causes  cognitive  dissidence,  the  verbal  cues  that  would 
be  noticed  are  in  the  “can’t  tell”  and  “don’t  know”  categories.  Although,  initially 
both  studies  looked  at  the  forget  category  as  a  verbal  cue  for  retrieval  difficulties, 
after  data  analysis,  it  turned  out  to  be  insignificant.  (Bolton  1993,  pp.  6-7,10-11) 

After  conducting  extensive  data  analysis  on  the  different  coding 
from  the  two  pilot  tests,  Bolton  determined  that  the  coding  categories  that  had 


32 


been  established  for  the  four  dimensions  of  cognitive  difficulties  (comprehension, 
retrieval,  judgment,  and  response)  were  all  accurate  for  both  sets  of  pilot  tests 
even  though  the  populations  were  different  (business  and  residential  customers). 
These  results  showed  that  automatic  coding  schemes  can  be  used  successfully 
for  evaluating  or  pretesting  surveys. 

This  thesis  examines  the  questionnaire  in  regards  to  the  four 
stages  of  macroprocesses  introduced  by  Touraneau  and  the  verbal  and  some  of 
the  non-verbal  cues  identified  in  the  GTE  study.  Through  the  protocol  analysis 
that  is  conducted,  the  survey  is  improved  by  looking  for  areas  of  cognitive 
dissidence  similar  to  the  GTE  study. 

3.  Avoiding  Problems 

There  are  many  problems  evaluators  can  encounter  when  developing 
surveys.  The  problems  that  are  most  common  relate  to  administering  the  survey, 
and  survey  formulation.  Researchers  must  struggle  with  the  following  potential 
problem  areas:  low  response  rates;  ensuring  honest  responses;  determining 
adequate  sample  sizes;  timing  of  surveys  and  wording  pitfalls.  (Nowak  1990, 
Johnson  1993) 

a.  Low  Response  Rates 

The  response  rate  of  a  survey  is  the  percentage  of  the  total  sample 
that  was  given  the  survey  who  respond.  A  “good”  response  rate  is  only  about  50 
percent.  (Nowack  1990,  P.146)  There  are  many  ways  to  increase  survey 
response  rates.  If  the  survey  is  conducted  internally  to  the  company,  the 
evaluator  can  provide  notice  ahead  of  time  that  a  questionnaire  will  be  sent  out. 
Employees  can  also  be  rewarded,  using  either  intrinsic  or  extrinsic  rewards,  or  a 
combination  of  the  two,  to  increase  response  rates.  Return  of  the  questionnaires 
should  be  made  easy  for  the  respondent.  If  it  is  an  internal  survey,  drop  boxes  or 
internal  mail  can  be  used.  If  the  survey  is  external,  prepaid  addressed  envelopes 
can  be  included  or  the  results  can  be  faxed  or  emailed  to  the  evaluator.  The 
questionnaire  should  also  look  professional  and  should  be  fairly  short.  “In 
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general,  as  the  length  of  any  questionnaire  increases,  its  reliability  increases  and 
the  compliance  rate  decreases.”  (Nowack  1990,  pp.  147-1 48) 


To  further  increase  the  response  rate,  evaluators  can  follow  up  with 
employees,  reminding  them  of  the  questionnaire.  Feedback  is  also  vital  when 
eliciting  responses,  especially  if  the  survey  is  internal  to  the  organization.  If 
surveys  are  often  given  but  results  are  never  provided,  employees  will  not  see 
the  value  of  the  surveys  and  will  not  see  that  their  opinion  is  actually  used. 
(Nowack  1990,  pp.147-148) 

b.  Anonymity  and  Honest  Responses 

Often  organizations  force  employees  to  respond  by  having  them 
sign  the  questionnaire.  While  this  does  increase  response  rates,  there  are  other 
problems  that  arise.  “When  a  survey  is  anonymous,  it’s  difficult  to  get  a  high 
return  rate.  When  it  is  not  anonymous,  the  reliability  of  answers  is  questionable 
because  employees  are  afraid  to  tell  the  truth.”  (Johnson  1993,  P.139)  This  has 
become  less  of  a  problem  with  technology,  enabling  people  to  take  a  survey 
anonymously  on  line;  however,  respondents  often  worry  that  their  identity  can  be 
determined  through  the  demographic  information  collected.  While  asking  for 
demographic  information  might  deter  some  respondents,  it  is  necessary  to  have 
to  understanding  the  data. 

When  conducting  a  survey  it  is  important  that  responses  are 
honest.  Unfortunately,  there  is  no  way  to  guarantee  honest  responses. 
Kirkpatrick  states  that  although  sometimes  knowing  the  identity  of  a  respondent 
is  useful,  it  is  important  to  refrain  from  asking  participants  to  sign  their  forms  in 
order  to  ensure  honest  reactions.  (Kirkpatrick  1998b,  p.34)  This  is  especially 
true  for  internal  organizational  surveys.  According  to  Kirkpatrick,  it  is  often  easier 
for  a  person  attending  an  outside  program  to  give  honest  feedback.  One  way 
around  this  is  to  include  a  space  for  the  respondent’s  name  on  the  reaction 
sheets  with  the  word  “optional.”  With  the  optional  name  space,  the  respondent 
can  make  their  own  decision  as  to  whether  to  include  his  or  her  name. 
Generally,  if  people  leave  positive  comments,  they  are  more  likely  to  include  their 
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names,  than  when  leaving  negative  comments.  If  a  respondent  does  include 
their  name,  they  can  be  quoted  if  they  say  something  positive.  In  addition,  this 
allows  for  the  evaluator  to  contact  the  person  with  further  questions  or 
clarification.  (Kirkpatrick  1998b,  p.34) 

c.  Minimum  Sample  Sizes 

It  is  not  always  necessary  to  survey  the  entire  population;  often  a 
sample  of  the  target  population  is  enough.  When  a  population  is  large,  using  a 
segment  of  the  population  is  much  less  expensive  and  often  generates  results  as 
reliable  as  surveying  the  entire  population.  However,  this  does  not  hold  true 
when  the  population  is  small.  (Paul  and  Bracken  1995,  p.  149)  There  are  several 
determining  factors  that  must  be  included  in  calculating  the  minimum  sample  size 
for  a  survey.  The  evaluator  must  also  consider  how  precise  the  population 
estimate  is  and  what  confidence  level  is  desired  in  the  results.  For  example, 
does  management  want  the  results  with  90  percent  accuracy,  95  percent,  or  99 
percent  accuracy?  The  smaller  this  margin  of  error  is,  or  the  greater  the 
confidence  desired,  the  larger  the  sample  size  has  to  be.  For  a  survey  to  be 
relevant,  the  sample  must  be  random  and  representative  of  the  entire  population. 
(Nowack  1990,  P.146)  Included  below  in  Figure  7  is  a  chart  of  minimum  sample 
sizes  for  target  population  totals. 
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Minimum  sample  sizes 


Table  for  determining  the  minimum 
sample  size  required  to  generalize  results 
to  apply  to  an  entire  population.  Level  of 
confidence  is  S5  percent  that  the  ample 
proportion  will  be  within  plus  or  minus  .05 
of  the  population  proportion.  Sample  size 
refers  to  questionnaires  returned,  not 
those  sen!  out.  Assuming  a  50  percent 
return  rate,  send  out  at  least  twice  the 
minimum  sample  sze. 


Population 

Sample 

Size 

Size 

10 

10 

50 

44 

100 

80 

200 

132 

300 

169 

400 

196 

500 

217 

Figure  7:  Minimum  Sample  Sizes  (From:  Nowack  1990) 

d.  Measurement  Timing 

As  Kirkpatrick  alludes  to,  there  is  no  “right”  time  to  evaluate  training; 
the  timing  depends  on  the  organization,  the  person,  and  the  training.  (Kirkpatrick 
1998b,  p.50)  Trainers  and  evaluators  do  not  agree  on  when  this  “ideal”  time  is  to 
evaluate  the  transfer  of  training.  There  are  arguments  for  measuring  this  transfer 
immediately,  or  one  to  12  months  later.  “Generally,  it’s  appropriate  to  measure 
the  initial  transfer  of  training  3-12  months  after  training,  with  six  months  being  the 
most  common  time  frame.”  (Garavaglia  1993,  p.  75)  Experts  also  recommend 
re-measuring  training  transfer  at  intervals  following  the  training  to  see  how  much 
of  the  material  is  still  retained  and  used.  Some  evaluators  recommend 
reevaluation  at  six-month  and  yearly  intervals.  (Garavaglia  1993,  p.  75) 
According  to  John  Newstrom  in  Transfer  of  Training,  “40  percent  of  skills  learned 
in  training  are  transferred  immediately,  25  percent  remain  after  six  months,  and 
only  15  percent  remain  a  year  later.”  (Garavaglia  1993,  p.76) 
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e.  Wording  Pitfalls/Writing  Techniques 

There  are  six  important  principles  to  follow  when  developing  survey 
questions:  Keep  the  survey  questions  aligned  with  the  goals  of  the  survey;  keep 
it  short;  avoid  double-barreling;  be  clear  and  concise;  avoid  leading  and  biased 
questions;  and  ensure  item-scale  agreement.  (Church  and  Waclawski,  1998 
p.77)  Most  of  these  are  self  explanatory,  but  there  are  a  few  concepts  that 
should  be  explained  in  more  detail.  Double-barreling  is  asking  more  than  one 
question  in  the  same  item.  This  can  pose  problems  because  often  it  is  difficult 
for  the  reader  to  interpret  the  question  and  similarly  for  the  evaluator  to  interpret 
the  answers.  Double-barreling  often  occurs  when  survey  designers  attempt  to 
shorten  the  length  of  the  questionnaires.  Another  prevalent  issue  is  leading  and 
biased  questions.  If  a  question  or  the  answer  choices  are  poorly  formed,  the 
question  could  lead  or  bias  the  responder  towards  a  certain  response.  Finally, 
ensuring  that  the  scale  used  matches  the  type  of  question  asked  is  very 
important.  (Church  and  Waclawski  1998,  pp. 77-83) 

When  using  such  scales  to  assess  people’s  opinions  in  a  positive 
or  negative  direction,  the  evaluator  has  to  determine  how  many  points  to  include 
on  the  scale.  Scaled  questions  can  be  either  even  or  odd  numbered,  and  usually 
range  from  four  to  nine  points.  (Paul  and  Bracken  1995,  p.  150)  One  pitfall  with 
using  an  odd  number  of  responses  is  it  might  lead  respondents  to  select  the 
middle  choice,  encouraging  a  neutral  response.  The  odd  number  scale  often 
causes  ambiguity  in  the  results.  One  study  found  that  if  a  question  gives  a 
neutral  option,  20  percent  of  respondents  chose  it.  (Paul  and  Bracken  1995,  p. 
150)  “An  even  number  of  response  options  with  no  neutral  midpoint  tends  to 
force  respondents  to  take  a  stand,  though  people  with  strong  attitudes  typically 
give  the  same  ratings  with  or  without  a  neutral  midpoint.”  (Paul  and  Bracken 
1995,  p.  150)  In  order  to  remove  this  ambiguity  and  to  force  an  opinion  out  of  the 
respondents,  the  neutral  or  midpoint  scale  has  been  removed  from  both  MSA 
surveys  when  feasible.  As  a  result,  most  of  the  degree  scales  the  author  chose 
had  four  response  options  with  no  neutral  midpoint. 
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D.  A  HISTORY  OF  STUDIES  ON  HUMAN  CAPITAL  INVESTMENTS 
1.  Value  of  Education  and  Training  to  the  Military 

a.  Changes  in  Performance 

A  thesis  by  Celik  (2002)  looked  at  the  relationship  between  job 
performance  of  DoD  civilian  employees  and  graduate  education.  The  data 
included  all  full-time  DoD  civilian  employees  from  1986  to  1999,  except  National 
Imagery  and  Mapping  Agency  employees  and  civilian  employees  outside  the  50 
states  and  the  District  of  Columbia.  The  data  was  obtained  through  the  Defense 
Manpower  Data  Center.  The  data  set  was  further  restricted  to  personnel  who 
possessed  at  least  a  B.A.  or  B.S.  and  were  part  of  the  General  Schedule  (GS)  or 
General  Management  (GM)  pay  scales.  In  all,  there  were  213,482  complete 
observations  in  the  data  set.  (Celik  2002,  p.29-30)  The  performance  measures 
used  were  salary,  promotion,  retention,  and  annual  performance  ratings.  The 
results  of  the  study  were  that  DoD  employees  with  a  Master’s  or  Doctorate 
received  higher  salaries  and  higher  performance  ratings  than  employees  without 
higher  education.  However,  these  people  are  more  likely  to  be  promoted  slower 
than  their  peers  because  they  start  off  at  higher  GS  pay  grades  and  there  is  less 
room  for  promotion  in  their  job  hierarchy.  In  addition,  Celik  (2002,  p.64)  found 
that  a  DoD  employees  with  a  Master’s  or  Doctorate  is  more  likely  to  leave  the 
service  earlier  due  to  increased  opportunities  outside  the  federal  service. 

b.  Impact  on  Promotion 

Little  research  has  been  conducted  to  tie  the  effect  of  graduate 
education  on  productivity.  (Bowman  and  Mehay  1999,  p.453)  Professor  Bowman 
of  the  US  Naval  Academy  and  Professor  Mehay  of  the  Naval  Postgraduate 
School  tried  to  do  just  that  in  an  attempt  to  determine  how  job  success  is  affected 
by  graduate  education.  The  study  looked  at  U.S.  Navy  officers  with  graduate 
degrees  and  their  promotion  rates  to  0-4  (Lieutenant  Commander).  The  study 
examined  data  on  all  officers  eligible  for  promotion  between  the  years  of  1985 
and  1990  from  the  Navy’s  Promotion  History  File.  In  total,  the  study  included 
6,583  line  and  staff  officers.  Bowman  and  Mehay  used  both  promotion  and 

supervisor  evaluations  as  performance  measures.  The  research  found  that 
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those  who  possess  a  graduate  degree  have  a  probability  of  promotion  10-15 
points  higher  than  those  without  an  advanced  degree.  The  probability  increases 
even  further  to  15-17  points  higher  for  individuals  who  received  their  degree 
through  a  full-time  funded  program,  such  as  NPS.  (Bowman  and  Mehay  1999,  p. 
460)  Bowman  and  Mehay  show  that  there  is  a  relationship  between  an 
individual’s  promotion  probability  and  graduate  education.  What  is  not  proven  is 
how  much  the  promotion  rate  is  affected  by  other  unobservable  attributes. 
Bowman  and  Mehay  suggest  that  some  of  these  unobservables  might  influence 
who  attends  or  is  selected  for  a  master’s  degree  program. 

Bowman  and  Mehay  pose  the  question  of  whether  students  are 
selected  based  on  job  performance  or  are  they  “sorted”  by  the  firm  or  leadership? 
Although  these  are  good  questions  and  the  direct  effect  graduate  education  on 
job  performance  has  not  been  demonstrated  conclusively,  there  is  undeniably  a 
relationship  between  the  two. 

c.  The  Value  of  Graduate  Education 

A  2002  study  by  Branstetter  measured  the  value  of  a  curriculum  at 
NPS  for  Marine  Corps  Officers.  The  data  he  used  represented  Marine  Corps 
officers  with  the  9648  Military  Occupational  Specialty  (MOS)  received  through  the 
Special  Education  Program  (SEP)  as  far  back  as  1987.  This  is  the  subspecialty 
that  is  earned  after  completing  the  Information  System  Technology  curriculum  at 
NPS.  A  population  list  was  obtained  from  the  Marine  Corps  Manpower 
Management  and  Officer  Assignments  database.  The  study  included  a  survey 
distributed  to  a  total  of  84  individuals.  (Branstetter  2002,  pp.  27,  31 )  A  total  of  44 
officers  returned  the  surveys,  for  a  survey  response  rate  of  52  percent;  however, 
due  to  incompleteness,  the  actual  response  rate  was  just  over  34  percent. 

In  determining  value  of  the  curriculum,  Branstetter  examined  many 
aspects  of  value.  He  looked  at  whether  the  curriculum  met  the  requirements  set 
by  the  Marine  Corps  and  the  usefulness  of  the  topics  based  on  responses  from 
graduates.  In  the  survey,  respondents  were  also  questioned  about  the  value  of 
their  thesis,  preparedness  for  follow-on  billets,  and  to  what  extent  they  used 
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different  knowledge  and  skills  that  were  taught  in  the  curriculum.  Respondents 
were  also  asked  additional  questions  such  as  how  much  they  liked  a  topic,  and 
the  value  of  the  “social  connection”  formed  at  NPS.  (Branstetter  2002,  pp.  43- 
56) 

In  determining  the  effect  of  personal  preference  on  the  perceived 
value  of  the  curriculum,  Branstetter  hypothesized  that  the  value  of  education 
would  vary  by  each  person’s  perspective.  “A  student’s  personal  preferences  may 
influence  both  the  student’s  and  the  organization’s  assessment  of  the  education 
value.”  (Branstetter  2002,  p.  52)  The  study  determined  that  individuals  give  a 
higher  value  to  topics  that  they  are  interested  in.  For  example,  Branstetter  (2002, 
p.53)  determined  that  there  was  an  89  percent  chance  that  a  topic  would  be 
given  a  rating  as  ‘valuable’  if  they  were  very  interested  in  the  topic. 

In  determining  the  value  of  the  peer  network  formed  at  NPS, 
respondents  were  asked  how  critical  these  relationships  were  to  their  follow-on 
billets  and  how  often  they  use  these  connections.  The  study  found  that  the 
respondents  placed  a  high  value  on  the  social  relationship  formed  at  NPS,  with 
an  average  rating  of  4.7  on  a  seven  point  scale  (7  most  critical).  The 
respondents  reported  a  60  percent  likelihood  of  contacting  a  fellow  student  on  a 
weekly  basis.  (Branstetter  2002,  p.  56) 

Branstetter  (2002)  uses  the  Knowledge  Value  Added  (KVA) 
approach,  introduced  by  Housel  and  Bell  (2001),  to  further  estimate  the  value  of 
the  1ST  curriculum  to  the  Marine  Corps.  In  this  portion  of  the  research,  the 
author  conducted  telephone  interviews  for  three  types  of  subspecialty-required, 
follow-on  billets.  He  received  a  job  description  and  an  estimation  of  the  amount 
of  time  needed  to  teach  each  required  skill  or  knowledge  for  the  job.  He  asked 
the  survey  respondents,  to  estimate  how  much  time  was  spent  learning  a  topic. 
The  cost  factor  is  computed  using  the  salary  of  the  person  in  each  job.  This  is 
based  on  the  Military’s  salary  pay  scale  according  to  rank.  Using  this 
information,  Branstetter  was  able  to  compare  which  knowledge  and  skills  were 
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being  utilized  the  most  for  which  jobs  and  the  cost  of  that  knowledge. 
(Branstetter  2002,  p.  62-67) 

Bowman  and  Mehay  (2004)  conducted  an  additional  study  of  the 
return  on  investment  for  Navy  graduate  education.  The  study  examined  the 
differences  in  career  progression  between  fully  funded  graduate  education,  off- 
duty  graduate  education,  and  no  graduate  education.  This  2004  study  looked  at 
all  Surface  Warfare  Officers  (SWO)  from  the  rank  of  Lieutenant  (0-3)  to  Captain 
(0-6).  The  data  included  all  officer  year  groups  from  1977  to  1989  and  included 
promotion  board  results  from  1981  to  2000.  Overall,  the  sample  size  was  over 
33,000.  Bowman  and  Mehay  conducted  a  cost-benefit  analysis  of  the  three 
different  graduate  education  alternatives.  The  study  identified  six  possible 
benefits  of  graduate  education  to  the  Navy:  increased  retention;  increased  labor 
productivity;  increased  unit  productivity;  faculty  and  student  research  output; 
technology  implementation;  an  increased  quality  of  life,  and  improved  job  and 
overall  satisfaction.  The  cost-benefit  analysis  conducted  looked  primarily  at 
retention  and  on-the-job  productivity  benefits.  To  determine  the  productivity 
benefits,  the  researchers  used  wage  differentials  in  the  civilian  labor  market  for 
individuals  with  master’s  degrees.  Because  only  two  benefits  were  discussed  in 
the  study,  Bowman  and  Mehay  state  that  the  benefits  of  graduate  education  are 
estimated  conservatively.  The  cost-benefit  analysis  estimated  the  cost  of  fully 
funded  graduate  education  at  $66.8  million  and  off-duty  education  at  $14.5 
million  (through  tuition  assistance).  Bowman  and  Mehay  found  that  funded 
graduate  education  had  a  large  retention  benefit  as  compared  to  the  ‘no  graduate 
degree’  alternative.  They  found  the  return  on  investment  was  1208  percent  on 
funded  education  versus  no  education.  However,  the  ROI  of  funded  versus  off- 
duty  education  was  only  3.8  percent.  This  means  that  the  study  found  funded 
education  has  a  3.8  percent  return  on  investment,  based  mostly  on  retention 
benefits,  as  compared  to  off-duty  graduate  education.  When  determining  the  net 
benefits  of  productivity,  the  researchers  again  found  that  fully-funded  graduate 
education  had  higher  net  benefits  than  off-duty  education.  Using  a  conservative 
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rate  of  return  of  two  percent  derived  from  the  civilian  labor  market,  Bowman  and 
Mehay  found  that  Navy  funded  graduate  education  has  a  6  percent  return  on 
investment  as  compared  to  off-duty  graduate  education  programs.  The 
conclusion  of  the  study  is  that  there  is  a  positive  net  benefit  of  funded  graduate 
education  over  off  duty  education.  (Bowman  and  Mehay  2004) 

One  limitation  of  the  study  is  that  only  Surface  Warfare  Officers 
were  included  in  the  analysis.  However,  it  is  expected  that  these  results  could  be 
duplicated  in  the  1200  community  (and  other  RL  and  staff  communities).  The 
selection  of  graduate  education  degrees  is  not  as  restrictive  as  in  the  SWO 
community.  If  graduate  education  was  only  offered  in  areas  that  can  be  used 
directly  in  a  community,  subspecialty  utilization  would  increase,  which  would 
directly  increase  on-the-job  productivity. 

A  1997  thesis  by  Eckardt  established  a  method  for  determining  the 
marginal  cost  of  a  student  at  NPS.  To  determine  the  cost  per  student,  the  author 
used  two  models.  The  first  model,  the  “Cost  per  Curriculum  Model”  can  be  used 
to  determine  the  average  cost  per  student  to  attend  NPS  for  FY96,  but  not  the 
marginal  costs.  The  data  used  in  adopting  the  first  model  was  obtained  through 
the  Registrar’s  Office  at  NPS,  using  FY96  reports.  (Eckardt  1997,  p.26)  The 
models  developed  allow  the  user  to  account  for  the  type  of  class  offered 
(residence  or  distance  learning  and  special  operations  and  international 
courses),  class  size,  number  of  lecture  hours,  and  number  of  laboratory  hours. 
The  second  model  is  used  to  determine  the  marginal  cost  per  student.  This 
model  is  developed  using  the  FY96  excess  capacity  at  NPS.  The  model 
developed  by  Eckardt  is  applied  to  the  Financial  Management  (FM)  Curriculum. 

Eckardt’s  research  indicates  that  there  is  a  large  variation  of  costs 
per  students  in  the  different  curricula  at  NPS.  Using  the  first  model,  Eckardt 
determined  that  the  average  cost  per  student  was  $15,391  in  FY96.  However, 
this  amount  varied  drastically  by  curriculum,  for  the  FM  curriculum  the  average 
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cost  per  student  was  $10, 884. 5  (Eckardt  1997,  p.84)  The  annualized  marginal 
cost  per  student  decreases  as  the  number  of  students  increases.  For  example, 
the  marginal  cost  of  adding  five  more  students  was  $11,947  in  FY96.  If  25 
students  were  added  to  enrollment,  the  annual  marginal  cost  would  be  only 
$7,500.  (Eckardt  1997,  p.85) 

The  FM  curriculum  is  in  the  same  department  as  the  MSA 
curriculum,  therefore,  it  can  be  assumed  that  some  of  the  costs  incurred  are  the 
same  or  at  least  similar.  The  most  obvious  difference  between  the  two  curricula 
is  length  of  the  programs.  The  FM  curriculum  is  an  18-month  program, 
compared  to  the  21 -month  MSA  curriculum.  Like  the  FM  curriculum,  the  MSA 
curriculum  also  starts  twice  a  year  therefore  three  different  sections  of  students 
are  enrolled  at  any  one  time.  Because  of  the  three  additional  months,  it  is  likely 
that  the  cost  per  student  and  the  marginal  cost  of  the  MSA  curriculum  is  greater 
than  that  of  the  FM  curriculum.  Although  the  computations  are  not  specifically  for 
the  MSA  curriculum,  Eckardt’s  research  provides  a  good  baseline. 

In  a  2005  study  conducted  for  the  Provost  of  the  Naval 
Postgraduate  School,  CDR  Hatch,  USN,  (Retired),  looked  at  how  the  MSA 
curriculum  at  NPS  was  aligned  with  the  needs  of  the  Navy.  The  data  set 
included  297  officers  with  3130  subspecialties.  There  were  officers  from  the 
Unrestricted  Line  community,  Restricted  Line  Community  and  the  Staff  Corps, 
ranging  in  rank  from  0-2  to  0-6.  Hatch  used  the  Navy  Officer  Billet 
Classifications  (NOBC)  to  describe  the  skills  needed  for  each  billet  requiring  a 
3130  P-or  Q-subspecialty.  The  study  identified  six  manpower-related  skills  and 
17  additional  skills  that  identified  as  important  to  3130  billets.  The  skills  are  listed 
below  in  Table  1.  These  skills  are  essential  components  of  the  MSA  curriculum 
in  order  to  meet  the  needs  of  the  Navy.  The  curriculum  is  based  on  nine  MSA 
Educational  skill  Requirements  (ESR)  that  are  derived  from  these  identified  skills. 
A  list  of  the  MSA  curriculum  ESRs  is  shown  below  in  Table  2. 

5  This  includes  the  Direct  Teaching  salary  of  the  faculty  with  fringe  benefits,  the  military 
salary,  and  OPTAR/travel  money. 
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Manpower-Related  Skills 

Manpower  Authorizations 

Manpower  Change  Requests 

Manpower  Determination 

Manpower  End-Strengths 

Manpower  Management 

Manpower  Requirements 

Additional  Skills  Required 

Career  Motivation  and  Promotion 

Personnel  Orientation 

Coordinate  Joint  DoD  Agencies 

Plans  for  Personnel  Procurement 

Data  Analysis 

Policy  Analysis 

Education  and  Training 

Promotion  Actions 

Joint  Personnel  Plans 

Ouantitative  and  Oualitative  skills 

Liason,  supervise,  direct 

Review  Legislation 

Management 

Staffing  Criteria 

Manning  Documents 

Survey  Analysis 

Naval  Personnel  Utilization  Plans 

Table  1:  Essential  Skills  for  31 30-Coded  Billets  (From:  Hatch  2005) 


MSA  Educational  Skill  Requirements 

D 

Organization  and  Management 

E 

Budgeting  and  Financial  Controls 

E 

Automated  Data  Analysis 

4 

Management  Fundamentals 

m 

Advanced  Ouantitative  and  Oualitative  Analysis 

ED 

MSA  —  Fundamental  Concepts 

7 

MSA  —  Policy  Analysis 

8 

Joint  Military  Strategic  Planning 

ED 

Evaluation,  Innovation  and  Creativity 

Table  2:  MSA  Educational  Skill  Requirements  (From:  Hatch  2005) 

d.  Utilization  of  Graduate  Education 

In  a  thesis  on  follow-on  utilization  of  graduate  education  in  the 
Navy,  Brutzman  (1994)  analyzed  two  separate  data  sets,  a  cross-sectional  and  a 
longitudinal  one.  The  cross-sectional  study  consisted  of  officers  in  the  1993 
Officer  Master  File  to  look  at  sub-specialty  utilization.  With  this  data  set,  she 
looked  at  all  the  officers  together,  then  separately  by  gender,  then  by  designator. 
The  data  set  used  included  all  Navy  officers  on  active  duty  FY93  from  Ensign  to 
Admiral.  The  study  looked  at  the  Unrestricted  Line,  Restricted  Line  and  all  Staff 
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Corps  communities  except  Medical  Corps  officers,  Dental  Corps  officers  and 
Judge  Advocate  General  Corps  officers  due  to  differences  in  their  education 
timelines.  The  percentage  representation  by  community  in  this  data  set  was  78 
percent  Unrestricted  Line,  11.7  percent  Restricted  Line,  and  10.3  percent  Staff 
Corps  officers.  In  total,  39,745  officers  were  included  in  the  final  data  set. 
(Brutzman  1994,  pp. 20-21) 

In  addition,  Brutzman  (1994)  conducted  a  longitudinal  study  using  a 
Cohort  file  of  officers  commissioned  in  1980  to  look  at  utilization  and  retention. 
The  data  used  was  obtained  from  the  Navy  Officer  Master  File  annual  records  for 
a  total  of  12  years,  from  1981  through  1993,  with  the  exception  of  1983.  The 
same  officer  communities  were  included.  (Brutzman  1994,  pp. 26-27) 

Brutzman  (1994,  p.33)  determined  that  only  13  percent  of  the 
officer  communities  analyzed  had  fully-funded  graduate  education.  When  the 
Navy  assigns  personnel  to  billets  for  subspecialty  utilization,  usually  they  are 
assigned  to  a  billet  requiring  the  exact  subspecialty  they  received;  this  would  be 
called  a  “direct  fill.”  If  instead  the  person  is  detailed  to  a  billet  requiring  a  similar 
subspecialty  (switch  must  be  an  approved  substitute),  then  this  is  called  a 
“related  fill.”  Brutzman  (1994,  p.26)  includes  both  direct  fills  and  related  fills 
when  identifying  subspecialty  utilization. 

In  the  cross-sectional  study,  Brutzman  found  that  over  50  percent 
of  the  Navy  Officers  who  receive  fully-funded  graduate  education  (FFGE)  have 
difficulties  fulfilling  their  subspecialty  payback  tours.  (Brutzman  1994,  p.31) 
However,  overall,  she  determined  that  the  Navy  gets  a  good  return  on  investment 
for  graduate  education  in  regards  to  subspecialty  utilization.  In  the  1993 
inventory,  78.2  percent  of  the  FFGE  subspecialists  had  utilized  their  subspecialty 
in  a  payback  tour  within  the  DoD’s  two  tour  requirement.  (Brutzman  1994,  p.39) 
These  seemingly  contradictory  results  can  be  explained  by  designator.  The  URL 
community  has  problems  meeting  DoD’s  targeted  utilization  rate  but  the  overall 
Navy  statistics  are  so  high  because  the  RL  and  Staff  Corps  communities’ 

utilization  is  high  and  raises  the  overall  Navy  average. 
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Compared  to  the  FY93  cross-sectional  study,  the  1980  cohort  data 
showed  much  lower  utilization  rates.  The  communities  had  the  same  results,  with 
the  RL  community  and  Staff  Corps  having  much  higher  utilization  rates  than  the 
URL  community.  Brutzman  (1994,  p.51)  attributes  some  of  the  discrepancy 
between  data  sets  to  the  career  progression  of  the  1980  cohort  in  1993.  At  only 
13  years  of  service,  she  notes  that  some  members  might  not  have  had  time  to 
complete  their  payback  tour  and  the  two-tour  period  would  not  have  expired  at 
the  end  of  the  observed  period. 

Looking  at  retention  rates,  Brutzman  determined  that  the  retention 
rate  for  officers  with  fully-funded  graduate  education  (FFGE)  is  higher  than  their 
peers  without  fully-funded  education  such  as  NPS.  (Brutzman  1994,  p.52-53)  At 
the  end  of  the  13-year  period  of  the  longitudinal  data,  89.2  percent  of  the  FFGE 
subspecialists  were  still  on  active  duty,  compared  to  33.7  percent  of  other 
subspecialists  (officers  who  have  graduate  education  but  not  through  a  fully- 
funded  graduate  education  program)  and  only  25.7  percent  of  officers  who  had 
no  graduate  education.  (Brutzman  1994,  p.53)  Taking  into  account  the 
commitment  in  years  of  service  owed  after  FFGE,  Brutzman  determined  that  the 
retention  rate  was  still  double  that  of  the  non-FFGE  officers.  (Brutzman  1994,  p. 
53) 

2.  Value  of  Education  in  the  Corporate  World 

Unfortunately,  many  U.S.  companies  do  not  associate  training  and 
education  directly  with  monetary  benefits.  Gordon  (1996,  p.67)  states  that  unlike 
the  US,  during  times  of  economic  uncertainty,  our  foreign  competitors  increase 
the  amount  of  training  invested  in  their  employees  because  they  see  human 
resource  development  as  their  “most  important  national  asset.”  Gordon  argues 
that  companies  need  to  start  investing  more  on  their  most  valuable  resource, 
their  people.  “By  adopting  a  workforce  education  policy,  any  organization  can 
harness  a  proven  economic-value-added  concept:  Human  knowledge  equals 
profit.”  (Gordon  1996,  p.  67)  In  a  1993  study  of  12,000  U.S.  businesses, 
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conducted  by  the  U.S.  Department  of  Labor,  it  was  determined  that  less  than  half 
offered  formal  job-skills  training  for  their  workers.  (Gordon  1996,  p.  67) 

a.  Short  Supply  of  Educated  Workforce 

Because  of  the  lack  of  training  offered  by  U.S.  companies,  many 
high-tech”  manufacturers  are  turning  towards  other  countries  for  their  educated 
workforce.  The  labor  supply  for  educated  workers  is  now  coming  more  from 
China,  Korea,  Singapore,  Ireland  and  Holland  than  before.  (Gordon  1996,  P.68) 
With  the  talent  shortage,  many  companies  are  outsourcing  their  labor  to  other 
countries.  “Outsourcing  work  to  other  countries  is  an  increasingly  popular 
method  of  coping  with  the  [labor]  shortage.”  (Fitz-enz  2000,  P.  xiv)  This  poses  a 
large  problem  for  the  military  because  outsourcing  is  not  always  an  option. 
Although  the  military  does  accept  some  non-citizens  in  their  enlisted  ranks,  this 
number  is  very  limited  and  the  enlistees  must  have  a  working  visa.  The  military 
has  further  tried  to  outsource  using  contractors  and  by  civilianizing  many  jobs. 
For  the  most  part,  however,  outsourcing  is  not  a  viable  option  for  the  services 
when  protection  of  national  security  is  at  stake.  As  a  result,  there  are  only  limited 
jobs  that  can  be  outsourced  to  civilians.  Therefore,  the  military  needs  to  make  its 
own  investments  in  Human  Capital. 

b.  Increased  Productivity 

A  1995  U.S.  Census  Bureau  survey  polled  3000  U.S.  businesses 
with  20  employees  or  more.  The  study  found  that  education  has  the  greatest 
increase  in  productivity  over  both  an  increase  in  hours  worked  and  an  increase  in 
capital  stock  (buildings,  tools  and  machinery).  For  non-manufacturing  jobs,  a  10 
percent  increase  in  education  leads  to  an  11  percent  increase  in  productivity. 
This  increase  compares  to  only  a  6.3  percent  increase  in  productivity  from  a  10 
percent  increase  in  hours.  (Gordon  1996,  p.67)  This  is  shown  graphically  in 
Figure  8,  below. 
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Figure  8:  Factors  for  Increased  Productivity  (From:  Gordon  1996) 


E.  CHAPTER  SUMMERY 

Although  it  is  not  always  easy  to  capture  the  return  to  an  organization  for 
graduate  education,  the  need  for  measurement  has  been  established. 
Organizations  today,  especially  the  military,  have  to  work  much  harder  to  quantify 
the  results  education  and  training  have  on  the  bottom  line.  Often  the  outcome  of 
education  and  training  can  be  seen  in  readiness  and  manning  levels  as 
indicators  of  improved  personnel  and  organizational  performance.  (Mehay  2005) 
Several  different  approaches  to  evaluating  and  measuring  human  capital 
investments  have  been  presented.  Kirkpatrick’s  four-level  model  of  evaluation 
and  different  author’s  interpretations  and  adaptations  have  been  introduced. 
Additional  methods  for  capturing  the  return  to  an  organization  such  as  Phillips’s 
Return-on-lnvestment,  Knowledge  Value  Added  and  Economic  Value  Added 
were  also  reviewed.  An  explanation  of  the  methods  was  followed  by  several 
examples  of  how  performance,  promotion,  productivity,  and  retention  are  all 
affected  by  graduate  education  both  in  the  Department  of  Defense  and  the 
corporate  world.  Understanding  these  methods  of  evaluation  and  the 
examination  of  previous  studies  develops  a  framework  for  analysis  of  the  value  of 
the  MSA  curriculum. 
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III.  SURVEY  FORMULATION 


The  literature  review  identified  several  methods  that  can  be  used  for 
evaluating  training  and  education  investments  and  discussed  ways  the  return  on 
investment  can  be  measured  by  an  organization.  This  thesis  builds  on  this 
foundation  to  develop  two  surveys  that  can  be  used  by  the  Navy  to  assess  the 
perceived  value  of  the  Manpower  Systems  Analysis  (MSA)  curriculum.  This 
chapter  discusses  the  objectives  and  scope  of  the  surveys,  explains  the 
instrument  that  is  created,  and  addresses  the  process  for  validating  the  survey. 

A.  OBJECTIVES  OF  SURVEYS 

There  are  three  objectives  for  developing  and  distributing  the  two  surveys. 
The  principle  goal  of  the  research  is  to  develop  two  survey  instruments  that  can 
be  used  to  obtain  an  assessment  of  the  curriculum  on  a  recurring  basis.  This 
assessment  will  be  based  on  the  perceptions  of  graduates  and  customers  of  the 
MSA  curriculum.  The  customer  is  seen  as  the  Navy  commands  where  MSA 
graduates  work  following  graduation.  These  customer’s  needs  are  assessed 
through  the  Immediate  Supervisor  Survey.  Over  time,  the  surveys  will  provide  a 
method  for  determining  the  difference  is  between  the  knowledge,  skills  and 
abilities  that  are  taught  through  the  MSA  program  and  what  is  needed  for  the 
manpower  billets  the  graduates  fill  (the  customer  needs).  Determining  what  the 
delta  is  between  what  is  taught  and  what  is  required  will  allow  for  curriculum 
improvement  during  the  biannual  curriculum  reviews. 

A  secondary  goal  of  the  surveys  is  to  provide  another  avenue  for 
validating  the  existing  billet  structure.  The  survey  responses  could  provide 
justification  or  rationalization  for  any  changes  or  additions  to  the  3130 
subspecialty  requirements  of  existing  billets.  A  third  objective  of  the  surveys  is  to 
increase  awareness  of  the  curriculum.  If  the  questionnaires  are  distributed  on  a 
regular  basis,  the  curriculum  can  be  kept  up  to  date,  while  marketing  the  value  of 
a  3130-coded  officer.  The  revised  versions  of  the  Graduate  Survey  and  the 
Immediate  Supervisor  Survey  are  included  in  Appendix  A  and  B,  respectively. 
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B.  SCOPE 

The  scope  of  this  chapter  includes  a  description  of  the  MSA  curriculum 
and  an  understanding  of  the  subject  populations.  The  Manpower  Systems 
Analysis  Curriculum  at  the  Naval  Postgraduate  School  (NPS)  is  examined. 
There  are  two  subject  populations  for  which  the  surveys  are  designed: 
graduates  and  their  supervisors. 

1.  Manpower  Systems  Analysis  Curriculum 

There  has  already  been  some  description  of  the  curriculum  in  Chapter  I. 
However,  more  background  is  helpful  when  looking  at  the  survey  design.  The 
MSA  curriculum  has  gone  through  a  series  of  changes  over  the  years.  One  of 
the  constant  changes  is  the  course  load.  To  keep  up  with  the  changing  needs  of 
the  Navy,  the  curriculum  courses  are  adjusted  often.  Currently,  most  of  the  first 
three  quarters  are  spent  learning  basic  graduate  management  topics  in  the  “core 
classes.”  These  classes  teach  the  concepts  of  Accounting,  Ethics,  Finance, 
Economics,  Operations  Management,  Statistics,  Strategy  Making  and  Budgeting 
and  are  required  for  the  degree  awarded  which  is  the  Master’s  in  Business 
Administration  (MBA).  After  the  basics  are  covered,  the  MSA  students  continue 
on  to  their  MSA-specific  classes.  There  have  been  several  changes  in  the  matrix 
over  the  years,  but  the  basic  idea  of  core  and  MSA-specific  classes  remains  the 
same. 

There  are  nine  Educational  Skill  Requirements  (ESRs)  that  are  taught 
over  the  duration  of  the  21 -months  of  the  MSA  program.  A  table  of  the  ESRs  is 
included  in  Chapter  II,  Table  2.  The  ESRs  serve  as  an  instructional  guideline  for 
each  course  in  the  curriculum  and  help  guide  which  courses  should  be  included 
in  the  program.  Out  of  the  nine  ESRs,  there  are  five  that  are  covered  by  the  core 
classes  and  the  others  are  taught  mainly  through  the  Manpower-specific  classes. 
The  standard  MSA  class  matrix  can  be  seen  below  in  Table  3. 
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Core  ManagementClasses 

GB3010  Managing  for  Organizational  Effectiveness 
GB3012  Communication  for  Managers 
GB3013  Problem  Analysis  &  Ethical  Dilemmas 
GB3020  Fundamentals  of  Information  Technology 
GB3040  Research  Methods  and  Data  Analysis 
GB3042  Operations  Management 
GB3050  Financial  Reporting  and  Analysis 
GB3051  Cost  Management 

GB3070  Economics  and  the  Global  Defense  Environment 
GB4014  Strategy  Making 
GB4043  Business  Modeling  Analysis 
GB4052  Managerial  Finance 

GB4053  Mission,  Structure  and  Resource  Determination 
GB4071  Economics  and  Cost  Benefit  Analysis 
GBXXXX  MBA  Elective 
NW3285  National  Security  Decision  Making 
MSA-Specific  Classes 

MN21 1 1  Seminar  in  Manpower,  Personnel,  and  Training  Issues  I 

MN21 12  Seminar  in  Manpower,  Personnel,  and  Training  Issues  II 

MN31 1 1  Personnel  Management  Processes 

MN3760  Manpower  Economics  I 

MN4106  Manpower/Personnel  Policy  Analysis 

MN4110  Multivariate  Manpower  Data  Analysis  I 

MN41 1 1  Multivariate  Manpower  Data  Analysis  II 

MN4116  Society  of  Human  Resource  Management 

MN41 19  Navy  Manpower  Requirements  Determination 

MN4761  Applied  Manpower  Analysis 

OS4701  Manpower  and  Personnel  Models 

Table  3:  MSA  Matrix  (From:  NPS  2006) 

The  MSA-specific  ESRs  are:  Automated  Data  Analysis;  Advanced 
Quantitative  and  Qualitative  Analysis;  Manpower  Systems  Analysis  Fundamental 
Concepts;  and  Manpower  Systems  Analysis  Policy  Analysis.  The  Automated 
Data  Analysis  ESR  requires  that  students  be  proficient  in  data  manipulation, 
statistics,  and  exploratory  data  analysis.  Additionally,  the  graduate  should  have 
a  “thorough  understanding  of  the  manpower  information  systems”  and  be  able  to 
conduct  and  execute  analysis  of  manpower,  personnel  and  training  issues.  (NPS 
2006)  The  Quantitative  and  Qualitative  Analysis  ESR  requires  graduates  to  be 
able  to  apply  different  “organizational,  economic,  statistical  and  mathematical 
techniques  and  concepts  to  manpower  and  personnel  policies  and  issues.”  (NPS 
2006).  This  includes  econometric  and  multivariate  analysis  of  large-scale 

databases  and  survey  data.  The  MSA  Fundamental  Concepts  ESR  requires  that 
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the  graduate  understand  manpower,  personnel  and  training  (MPT)  concepts 
within  DoD  and  DoN.  The  manpower  issues  a  graduate  should  be  familiar  with 
are:  billet  authorizations  and  cost;  requirements  determination;  end  strength 
planning;  and  total  force  planning  and  programming.  The  personnel  issues  the 
MSA  curriculum  is  required  to  cover  include:  recruiting;  officer  and  enlisted 
community  management;  accession  plans  and  policies;  attrition  and  retention; 
compensation;  and  readiness.  The  training  areas  that  a  graduate  should 
understand  are:  theories  of  learning;  instructional  technologies;  the  systems 
approach  to  training;  training  cost  and  effectiveness  evaluation;  and  training  as  it 
relates  to  fleet  readiness.  The  policy  analysis  ESR  requires  that  a  graduate 
have  the  skills  necessary  to  analyze,  present  alternatives,  and  understand  the 
impact  of  manpower,  personnel  and  training  policies.  (NPS  2006) 

The  Graduate  School  of  Business  and  Public  Policy  (GSBPP)  is  one  of 
only  two  institutions  across  the  country  that  has  been  accredited  by  both  The 
Association  to  Advance  Collegiate  Schools  of  Business  (AACSB)  and  The 
National  Association  of  Schools  of  Public  Affairs  and  Administration  (NASPAA). 
(NPS  2006b)  Because  most  of  the  core  classes  are  required  for  accreditation  by 
these  two  accreditation  organizations,  and  the  goal  of  the  surveys  is  to  receive 
feedback  from  the  MSA  curriculum,  not  GSBPP,  the  questions  will  be  centered 
on  the  MSA-specific  ESRs  described  above. 

2.  Subject  Populations 

a.  The  Graduates 

The  intended  population  for  the  first  survey,  included  in  Appendix  A, 
is  the  pool  of  graduates  of  the  MSA  curriculum.  The  survey  is  designed  for  all 
Navy  graduates  as  opposed  to  just  1200  (HR)  officers.  Human  Resource  (1200) 
officers  make  up  a  sizable  portion  of  the  students  enrolled  in  the  MSA  curriculum, 
although  this  ratio  is  decreasing.  According  to  the  Strategic  Planning  and 
Analysis  Directorate  (N14),  the  percentage  of  1200  officers  to  other  communities 
represented  in  the  MSA  curriculum  will  continue  to  decrease  as  other 
communities  see  the  importance  of  the  MSA  curriculum  and  1200  officers 
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expand  into  other  curriculums  such  as  Human  Systems  Integration  (HIS)  and 
Operations  Research  (OR).  (W.  Wagner,  personal  communication,  May  5,  2006) 
Because  the  curriculum  is  not  designed  specifically  for  1200  officers,  the 
responses  from  graduates  of  other  communities  are  also  valued.  Therefore,  the 
instrument  is  constructed  to  elicit  responses  from  all  U.S.  Navy  graduates.  At 
this  time,  the  survey  is  intended  for  those  who  have  graduated  from  the 
curriculum  between  1996  and  2004.  This  time  period  restricts  the  survey  to 
officers  who  are  at  least  12-18  months  beyond  graduation.  This  restriction  allows 
for  the  graduate  to  have  some  operational  experience  prior  to  answering  the 
survey.  The  insight  an  experienced  graduate  offers  is  much  more  valuable  than 
someone  who  just  graduated  from  the  program.  The  experienced  officer  has  had 
time  in  to  serve  in  Manpower-related  billets,  enabling  him  or  her  to  see  what 
value  the  curriculum  added  to  their  follow-on  billets. 

b.  The  Supervisors 

The  customers  of  the  MSA  curriculum  are  the  commands  where 
graduates  are  working.  The  best  person  suited  to  evaluate  the  MSA  graduate  is 
usually  their  immediate  supervisor.  For  this  reason,  the  customer  survey  is 
designed  for  the  immediate  supervisor.  The  immediate  supervisors  of  MSA 
graduates  in  any  Manpower,  Training  and  Personnel  billets  could  either  be 
military  or  civilian.  The  military  supervisor  ranks  range  from  Lieutenant  (0-3)  to 
Rear  Admiral  (0-7).  The  civilian  supervisors  could  hold  the  position  of  General 
Schedule  (GS-10  to  GS-15)  or  Senior  Executive  Service  (SES).  As  of  2005, 
there  were  153  funded  billets  in  the  Navy  with  3130  P-or  Q-coded  primary 
subspecialty.  (Hatch  2005)  This,  however,  does  not  mean  that  the  population  is 
exactly  153  because  officers  without  the  3130-subspecialty  might  be  filling  the 
billet.  The  survey  is  intended  for  the  immediate  supervisors  of  the  3130  P-or  Q- 
coded  subspecialty  officers. 

C.  SURVEY  FOR  GRADUATES 

This  section  provides  an  explanation  of  the  types  of  questions  asked  in  the 

survey  of  graduates  of  the  MSA  curriculum.  Each  question  in  the  survey  can  be 

classified  into  one  of  four  sections:  (1)  demographic  questions;  (2)  NPS 
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education-related  questions;  (3)  MSA-specific  education  questions;  and  (4) 
additional  comments. 

1.  Demographic  Questions 

The  demographic  questions  are  required  for  statistical  analysis  and  are 
based  on  both  the  officer’s  military  and  graduate  education  experiences. 
Because  the  curriculum  has  changed  so  much  over  the  years,  the  month  and 
year  of  graduation  are  asked.  Also,  the  graduate’s  designator  is  requested.  This 
is  important  because,  for  example,  responses  might  differ  between  an 
Unrestricted  Line  Officer  and  a  Restricted  Line  Officer.  Also,  their  previous 
military  experience  might  affect  their  responses,  such  as  between  a  lateral 
transfer  from  another  community  and  a  direct  accession.  The  graduate  are 
asked  if  they  had  a  post-baccalaureate  degree  prior  to  entering  the  MSA 
curriculum.  Previous  graduate  education  could  affect  how  a  person  perceives 
the  value  of  the  curriculum.  Each  respondent  is  then  asked  to  list  the  time  spent 
in  Manpower,  Personnel,  and  Training  (3130  P-or  Q-coded)  billets.  This 
information  can  be  used  to  determine  the  amount  of  experience  the  graduate  has 
in  the  MPT  field.  Occasionally,  some  anomaly  in  the  results  can  be  explained 
through  descriptive  statistics.  Therefore,  to  facilitate  analysis  and  understanding 
of  the  results,  demographic  questions  are  essential. 

2.  NPS  Education-Related  Questions 

There  are  many  aspects  of  the  NPS  educational  environment  that  provide 

value  but  are  not  specific  to  the  MSA  curriculum.  Each  of  these  aspects  needs  to 

be  assessed  to  fully  capture  the  value  of  the  curriculum  to  the  graduate.  For 

example,  the  officer  is  asked  how  he  or  she  selected  their  thesis  topic.  The 

follow-on  question  is  for  the  respondent  to  rate  how  relevant  his  or  her  thesis  was 

to  their  MPT  billets  and  their  Navy  career.  The  questionnaire  also  includes  a 

question  to  evaluate  the  value  of  the  unique  environment  at  NPS.  Besides  the 

Navy  students  from  different  communities,  the  military  student  body  of  NPS 

consists  of  Army,  Air  Force,  and  Marine  Corps  Officers.  In  addition,  there  are 

DoD  civilian  employees  and  military  and  civilian  foreign  students.  There  are  few 

other  opportunities  to  interact  with  such  a  diverse  student  population.  In  his 
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thesis,  Branstetter  (2002)  found  that  a  majority  of  students  stated  peer 
relationships  as  critical  to  their  success  in  subsequent  follow-on  billets. 
(Branstetter  2002  p.56)  Therefore,  it  is  important  to  capture  the  effect  of  peer 
interactions  as  a  possible  source  of  value  in  the  survey. 

3.  MSA-Specific  Education  Questions  (ESRs) 

There  are  several  questions  that  are  designed  to  assess  the  value  of  the 
MSA  curriculum  based  on  the  MSA-specific  ESRs  described  above.  These 
questions  relate  directly  to  the  MPT  topics  taught  in  the  curriculum.  The 
respondent  is  asked  to  rate  the  extent  to  which  they  were  prepared  for  their 
follow-on  billets  via  the  different  MSA-specific  areas.  When  analyzing  the  results, 
if  there  is  one  area  or  one  specific  ESR  that  many  respondents  (and  their 
supervisors)  feel  is  lacking,  then  this  is  where  future  improvements  could  be 
made  in  the  curriculum. 

4.  Additional  Comments 

Although  close-ended  questions  allow  for  easier  interpretation,  often 
comment  sections  provide  greater  detail.  Allowing  the  respondent  to  suggest 
changes  or  make  any  additional  comments  augments  the  amount  of  qualitative 
information  that  can  be  gathered.  As  discussed  in  Chapter  II,  Kirkpatrick 
suggests  using  open-ended  questions  in  order  to  understand  the  reasons  for  how 
a  respondent  answers  a  question.  (Kirkpatrick  1998b,  p.25)  The  comment 
questions  in  the  graduate  survey  serve  this  purpose  and  also  allow  the 
respondent  to  share  anything  else  they  he  or  she  feels. 

D.  SURVEY  FOR  SUPERVISORS 

This  section  provides  an  explanation  of  the  types  of  questions  asked  in  the 
survey  of  the  immediate  supervisors  of  the  MSA  graduates.  Each  question  in  the 
survey  can  be  classified  into  one  of  three  sections:  (1)  demographic  questions; 
(2)  MSA-specific  education  questions;  and  (3)  additional  comments. 

1.  Command  Demographic  Questions 

The  demographic  questions  again  are  asked  in  order  to  provide  some 
background  for  the  survey  respondents.  There  is  a  possibility  that  the  responses 

might  be  reflective  of  certain  characteristics  of  the  population.  In  order  to  control 
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for  these  variables  in  the  data  analysis,  it  is  important  to  gather  some  basic 
demographic  information.  The  background  information  that  is  asked  pertains  to 
the  officer  who  is  answering  the  survey.  The  supervisors  are  asked  which  officer 
community  they  belong  to,  what  their  rank  is,  and  if  they  have  the  3130- 
subspecialty.  Each  of  these  questions  give  some  insight  into  the  supervisors’ 
experience  and  military  background.  The  supervisors  are  also  asked  how  many 
and  what  type  of  3130-coded  billets  they  supervise. 

3.  MSA-Specific  Education  Questions  (ESRs) 

This  section  attempts  to  determine  if  there  is  a  difference  between  MSA 
graduates  and  other  officers,  both  with  and  without  a  Master’s  degree.  The 
supervisors  are  asked  questions  to  determine  how  steep  the  learning  curve  is  for 
the  graduates  to  become  proficient  in  their  3130-coded  billets.  The  supervisors 
are  also  asked  about  the  graduates’  proficiency  level  in  certain  key  areas.  These 
key  areas  correspond  with  the  MPT-related  ESRs.  The  goal  of  this  section  of  the 
supervisor  survey  is  to  determine  if  the  graduate  possesses  the  MPT  skills 
necessary  to  succeed  in  a  3130-coded  billet. 

4.  Additional  Comments 

Similar  to  the  graduate  survey,  additional  comments  are  valued  for  further 
qualitative  information.  The  supervisors  are  asked  to  explain  their  answers  to 
some  questions  and  are  also  encouraged  to  provide  any  additional  feedback 
about  the  MSA  graduates.  This  additional  section  provides  the  supervisor  with 
an  avenue  to  address  anything  he  or  she  feels  is  relevant  that  was  not  discussed 
earlier  in  the  survey. 

E.  EVALUATING  SURVEYS 

If  an  evaluation  instrument,  such  as  the  surveys  developed,  is  intended  for 

repeated  use,  it  is  recommended  that  some  testing  prior  to  distribution  is 

conducted.  (Phillips  1997,  p.89)  As  covered  in  Chapter  II,  there  are  many 

methods  for  evaluating  and  validating  surveys  prior  to  their  distribution.  It  is 

important  to  review  the  questionnaire  for  some  of  the  pitfalls  discussed  in 

Chapter  II.  When  designing  the  survey,  the  questions  had  to  be  carefully  crafted 

not  to  introduce  bias  or  lead  the  results  toward  a  particular  answer.  Even  after 
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these  aspects  are  taken  into  consideration,  further  testing  had  to  be  conducted  to 
ensure  the  survey  would  be  suitable  for  the  sample  audience  and  would  elicit  the 
required  information. 

1.  Protocol  Analysis 

A  protocol  analysis  was  conducted  to  further  test  the  surveys.  The 
protocol  analysis  for  the  Graduate  Survey  was  conducted  using  both  current 
MSA  students  and  former  MSA  graduates.  The  protocol  analysis  for  the 
Supervisor  Survey  was  conducted  using  military  faculty  at  NPS  who  had 
previously  worked  in  MPT  billets,  and  select  military  faculty  outside  of  NPS  in 
current  MPT  supervisory  positions.  Subjects  were  asked  to  read  the  survey 
out-loud  while  the  session  was  recorded.  As  the  participants  read  the  survey 
questions,  they  were  asked  to  think  out-loud  as  well  and  verbalize  any 
hesitations  they  had.  As  with  the  protocol  analysis  conducted  by  Bolton  (1993), 
discussed  in  Chapter  II,  the  reactions  of  the  readers  were  observed.  As  the 
subject  read  the  question  out-loud,  the  observer  (the  author  in  this  case)  looked 
for  evidence  of  misunderstanding  or  confusion.  This  confusion  could  manifest  in 
the  form  of  hesitation  by  the  reader,  re-reading  the  question,  a  lengthy  pause  or 
asking  clarifying  questions  about  the  survey.  Following  the  protocol  analysis,  the 
surveys  were  further  revised  and  a  second  protocol  analysis  was  conducted  with 
the  revised  surveys,  using  a  different  sample  of  the  population.  The  protocol 
analysis  was  conducted  until  it  was  found  no  new  problems  or  changes  were 
found.  In  total,  ten  supervisors  and  ten  graduates  were  interviewed.  The  final 
product  after  the  protocol  analysis  is  the  revised  Graduate  and  Supervisor 
surveys  included  in  Appendix  A  and  B,  respectively. 

2.  Pilot  Testing 

Traditionally,  after  a  protocol  analysis  is  conducted,  the  survey  is  tested 

prior  to  distribution  to  the  target  audience.  This  preliminary  testing  is  called  either 

“pilot  testing”  or  “pretesting”  a  survey.  Phillips  recommends  conducting  a  pilot 

test  and  analyzing  the  data  to  see  if  any  problems  arise  with  the  evaluation 

instrument.  (Phillips  1997,  p.89)  There  are  several  characteristics  that  have  to 

be  evaluated  to  determine  if  an  evaluation  method  is  effective:  validity;  content 
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validity;  construct  validity;  concurrent  validity;  predictive  validity;  and  reliability. 
(Phillips  1997,  pp. 89-92)  Conducting  a  pilot  test  with  the  sample  audience 
provides  an  avenue  for  evaluation  of  the  instrument.  Pretesting  is  important 
because  often  the  survey  developers  miss  problems  with  the  instrument  because 
some  are  too  familiar  with  the  material.  The  developer  already  understands  what 
the  question  is  supposed  to  be  asking  and  might  not  realize  that  the  meaning 
might  be  unclear  to  an  outsider.  (Edwards,  Thomas,  Rosenfeld,  Booth-Kewley 
1997,  p.84)  Although  there  is  some  discussion  about  what  the  ideal  pretesting 
sample  size  should  be,  Edwards,  Thomas,  Rosenfeld,  and  Booth-Kewley, 
recommend  using  eight  to  12  people  per  pretesting  group.  In  their  opinion,  this 
allows  for  enough  different  viewpoints  but  is  a  small  enough  group  to  be  feasible 
in  a  short  time  period.  ((Edwards,  Thomas,  Rosenfeld,  Booth-Kewley  1997, 
P-85) 

For  the  purpose  of  this  thesis,  no  additional  pretest  was  conducted.  This 
decision  was  based  on  two  factors:  necessity  and  time.  The  feedback  that  was 
received  through  the  one-on-one  interviews  in  the  protocol  analysis  was 
extensive  and  more  valuable  than  tabulated  results  would  have  been.  Due  to  the 
qualitative  feedback  received  in  the  first  protocol  analysis,  a  second  set  was 
conducted  instead  of  a  pilot  test.  The  survey  was  revised  after  the  initial 
interviews  and  then  a  second  set  of  interviews  were  conducted.  Time  restriction 
was  the  second  reason  an  additional  protocol  analysis  was  conducted  in  lieu  of 
pretesting. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  objective  of  this  thesis  was  to  develop  a  way  to  capture  the  perceived 
value  of  the  Manpower  Systems  Analysis  (MSA)  curriculum  at  the  Naval 
Postgraduate  School  (NPS).  The  thesis  developed  two  surveys  that  can  be 
administered  to  graduates  of  the  MSA  curriculum  and  their  immediate 
supervisors.  This  last  chapter  discusses  some  of  the  conclusions  reached  in  the 
research  of  evaluation  methods  and  recommendations  for  further  study  in  the 
area  of  the  evaluation  of  training  and  education  investments. 

1.  Types  of  Metrics  to  Evaluate  Training  and  Education 

Several  different  methods  for  measuring  the  return  on  human  capital 

investments  were  discussed  in  Chapter  II.  Some  of  the  most  common  ways  to 
evaluate  training  and  education  include:  Kirkpatrick’s  four-level  model  of 
evaluation;  Phillips’  Retum-on-lnvestment  (ROI)  Model;  and  Housel  and  Bell’s 
Knowledge  Value  Added  (KVA)  approach.  There  were  also  several  different 
forms  of  instruments  that  can  be  used  to  conduct  these  evaluations.  Although 
this  thesis  chose  to  create  two  surveys,  the  process  of  identifying  the  perceived 
value  of  the  MSA  curriculum  could  have  also  been  conducted  through  a  series  of 
interviews  or  focus  groups.  The  author  chose  to  develop  two  surveys  because 
the  widespread  geographic  distribution  of  the  target  audience  would  make 
interviews  or  focus  groups  too  costly.  In  addition,  the  reaction  to  the  MSA 
curriculum  of  both  the  supervisors  and  graduates  is  an  important  aspect  of  the 
curriculum’s  value.  The  levels  of  satisfaction,  or  audience  reactions  are  most 
easily  measured  through  a  survey.  This  method  allows  for  honest  responses 
without  threat  of  reprisal  due  to  the  anonymity  of  the  survey  process. 

2.  Survey 

The  surveys  were  developed  and  then  revised  through  a  protocol  analysis. 
Through  these  series  of  one-on-one  interviews,  the  author  was  able  to  eliminate 
any  biased  or  leading  questions  and  able  to  reword  questions  that  proved  to  be 
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confusing  or  ambiguous.  The  curriculum  sponsor,  the  Strategic  Planning  and 
Analysis  Directorate  (N14),  will  be  able  to  determine  the  difference  between  what 
is  expected  in  an  MSA  graduate  by  their  supervisors,  and  what  the  graduate 
learns.  Through  repeated  distribution  of  the  surveys,  and  revision  of  the 
curriculum  from  the  results;  the  needs  of  the  Navy  will  be  constantly  assessed 
and  the  curriculum  adjusted  to  meet  these  evolving  needs.  The  two  surveys 
provide  an  avenue  for  this  feedback  from  both  the  graduates  and  the  supervisors 
of  Manpower  Systems  Analysis,  graduates  working  in  3130-subspecialty  code 
billets. 

3.  Protocol  Analysis 

Elements  of  a  protocol  analysis  were  discussed  in  Chapter  II,  and  Chapter 
III  discussed  specifically  the  Protocol  Analysis  that  was  conducted  in  this  thesis. 
The  Protocol  analysis  resulted  in  the  rewording  or  elimination  of  several  of  the 
original  survey  questions.  Some  of  the  changes  that  were  made  were  simply 
from  an  aesthetic  or  readability  standpoint.  Other  changes  were  due  to 
misinterpretations  of  the  questions.  This  occurred  when  the  author  intended  one 
question  to  be  asked  but  the  reader  understood  something  different.  Most  often 
this  resulted  in  rewording  of  the  question  but,  on  occasion,  the  question  was 
removed  if  several  people  experienced  problems  with  it.  Additional  protocol 
analyses  were  conducted  after  each  revision  of  the  surveys.  The  author  ceased 
to  revise  the  surveys  once  the  Protocol  Analyses  process  no  longer  produced 
changes  to  be  made. 

4.  MSA  Utilization 

Other  studies  have  examined  utilization  of  graduate  education  by 
designator  but  no  research  has  been  conducted  specifically  on  the  MSA 
curriculum.  Through  the  questionnaires,  the  resource  sponsor,  N14,  will  be  able 
to  assess  the  utilization  of  3130  P-and  Q-coded  personnel.  The  surveys  will 
provide  another  method  for  examining  the  existing  billet  structure.  An  analysis  of 
the  results  will  indicate  if  the  requirements  for  billets  requiring  the  3130  P-and  Q- 
coded  subspecialty  reflect  the  actual  placement  of  MSA  graduates. 
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B.  RECOMMENDATIONS 

This  thesis  only  utilizes  one  method  for  determining  value  of  the  MSA 
curriculum.  Additional  evaluation  would  lead  to  a  more  thorough  estimation  of 
value.  There  are  four  additional  areas  of  study  that  are  recommended  for  future 
analysis.  The  recommendations  are  based  on  adding  additional  survey 
questions,  adding  additional  target  audiences,  looking  at  different  methods  to 
measure  value,  and  related  topic  research. 

1.  Additional  Survey  Questions 

In  an  effort  to  keep  the  survey  brief  and  to  increase  the  response  rate 
some  potentially  valuable  questions  were  omitted.  Branstetter  (2002,  p.52) 
determined  that  an  individual’s  personal  preference  for  a  subject  increases  their 
perception  of  the  value  of  that  topic.  Because  of  this  finding,  it  is  recommended 
that  the  future  graduate  survey  incorporate  a  “liking”  factor.  The  series  of 
questions  that  would  be  required  for  this  evaluation  would  have  made  the  survey 
in  this  thesis  much  longer  than  desired.  One  question  was  included  to  capture 
this  information  for  the  entire  curriculum  by  asking  the  graduate  the  method  by 
which  they  were  selected  to  attend  NPS.  Additional  value  would  be  added  if  the 
graduate  were  asked  how  much  they  liked  or  enjoyed  each  specific  topic  within 
the  curriculum.  A  comparison  of  interest  in  the  topic  to  their  perceived  value  of 
each  topic  might  help  to  explain  some  of  the  results. 

Previous  experiences  of  both  the  graduates  and  supervisors  also  might 
influence  how  the  surveys  are  answered.  To  fully  understand  the  answers  it 
might  be  beneficial  to  add  a  series  of  questions  to  establish  previous  billets  held 
and  experience  level  of  the  respondents.  Some  questions  were  included  to 
ascertain  some  of  the  background  of  respondents  for  both  the  supervisor  and 
graduate  surveys.  The  supervisors  were  asked  to  identify  their  experience  level 
in  Manpower,  Personnel  and  Training  (MPT)  billets  and  to  indicate  their 
designator.  Graduates  were  also  asked  if  they  had  laterally  transferred  and,  if 
so,  what  their  previous  designator  was.  For  graduates  it  would  be  useful  to  know 
additional  information  about  their  tours  prior  to  receiving  graduate  education. 


61 


Some  of  this  information  is  derived  from  their  designators,  but  the  extent  of  their 
experience  or  what  type  of  billets  they  held  is  not.  For  example,  if  the  graduate 
worked  in  a  MPT  billet  prior  to  enrolling  in  the  MSA  curriculum,  his  or  her 
previous  experience  might  improve  the  rate  of  understanding  of  the  material.  For 
supervisors,  the  extent  of  their  MPT  experience  might  not  be  obvious  just  by 
asking  their  subspecialty  code.  Also,  if  the  officer  changed  designators,  again 
their  experiences  might  shape  their  responses  differently. 

As  previously  mentioned,  these  additional  questions  were  not  included  in 
the  surveys  at  this  time  in  order  to  limit  the  length  of  the  survey.  In  the  future,  if 
analysis  of  the  survey  results  leads  to  any  unexpected  or  unexplained  results,  the 
surveys  could  be  revised  to  include  these  additional  questions.  While  these 
additional  questions  are  not  necessary  at  this  time,  they  might  yield  insight  into 
the  answers  of  subsequent  surveys. 

2.  Audience  Recommendations 

In  addition  to  adding  more  questions  to  subsequent  surveys,  the  target 
audiences  could  be  expanded.  The  audience  recommendations  relate  to  the 
graduate  survey  and  to  additional  survey  possibilities. 

a.  Graduates 

Currently,  the  graduate  survey  is  targeted  only  for  active  duty  Navy 
officers  who  graduated  from  the  curriculum  between  1996  and  2004.  To 
increase  the  population  size  and  allow  for  more  feedback  on  the  curriculum,  this 
target  could  be  expanded.  The  date  of  graduation  could  be  expanded  to  include 
graduates  six  months  after  graduation.  According  to  Kirkpatrick  (1998b,  p.55), 
some  people  change  immediately  and  others  take  up  to  six  months  to  adapt  what 
they  learned.  Waiting  six  months  after  graduation  allows  the  graduates  sufficient 
time  to  become  acquainted  with  their  jobs,  but  is  still  recent  enough  to  gain 
valuable  recent  feedback.  Several  questions  are  asked  of  the  graduates  to 
determine  graduation  date  and  to  gauge  MPT  experience.  After  the  results  are 
collected,  the  demographic  information  could  be  used  to  further  divide  the  data 
into  separate  demographic  categories,  such  as  the  length  of  time  since 
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graduation.  This  would  allow  for  the  results  to  be  excluded  if  it  was  determined 
that  the  new  graduates  are  too  inexperienced  for  the  purpose  of  the  study. 


There  are  several  ways  to  change  the  target  population  for  the 
graduate  survey.  The  survey  could  be  developed  for  all  MSA  graduates, 
including  non-Naval  and  international  students.  This  type  of  survey  would  need 
to  be  rather  lengthy  and  it  could  be  very  difficult  to  find  contact  information  for 
everyone,  especially  the  international  students.  A  more  feasible  change  would 
be  to  survey  all  US  students,  including  each  Service  and  DoD  civilian  students. 
Another  recommendation  would  be  to  develop  a  survey  for  all  US  students 
serving  in  MPT  billets.  This  would  provide  the  most  relevant  responses.  Adding 
this  additional  audience  would  require  a  longer  survey  and  more  details  on  the 
Marine  Corps’  follow-on  billets. 

Another  additional  possible  audience  for  the  graduate  survey  would 
be  to  include  retired  and  separated  graduates.  The  survey  could  be  adjusted  to 
include  how  relevant  the  MSA  curriculum  has  been  to  their  civilian  career. 
Adding  this  additional  segment  to  the  target  population  could  provide  some 
valuable  retention  information.  To  target  this  segment  of  the  population,  it  would 
be  preferential  to  develop  a  separate  survey  than  the  one  administered  to  active 
duty  graduates,  to  limit  the  length  of  both  surveys.  Unfortunately,  it  is  difficult  to 
find  contact  information  for  personnel  who  have  separated  from  the  military; 
therefore,  responses  might  be  limited  to  those  who  joined  the  reserves. 
However,  the  reserves  represent  a  highly  selected  population  and  restricting  the 
group  in  this  way  could  bias  the  results,  because  it  does  not  portray  the 
preferences  of  the  entire  separated  population. 

b.  Additional  Survey 

According  to  Kirkpatrick  (1998b,  p.1),  the  supervisor  is  not  always 
the  best  person  to  conduct  an  evaluation.  Other  individuals  in  the  workplace 
might  also  be  ideal  evaluators.  Using  the  360-degree  evaluation  theory, 
additional  surveys  could  be  developed  to  provide  a  more  rounded  observation  of 
the  graduate.  For  example,  A  survey  could  be  developed  for  peers  of  the  MSA 
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graduates.  This  would  provide  an  evaluation  from  another  aspect  of  the 
organization. 

3.  Alternate  Evaluation  Methods 

Two  surveys  are  developed  by  this  thesis  to  evaluate  the  MSA  curriculum. 
Several  additional  methods  for  evaluating  training  and  education  effectiveness 
are  introduced  in  Chapter  II  by  Carnevale  and  Schulz  (1990,  p.246).  Due  to  the 
geographic  distribution  of  the  target  audience,  surveys  are  a  more  feasible  way 
to  get  a  large  response.  According  to  Kirkpatrick  (1998b,  p.52),  interviews  are  a 
great  method  for  obtaining  additional  information  yet  are  very  time  consuming.  If 
additional  resources  and  time  are  available,  interviews  could  also  be  very  helpful 
in  measuring  value  of  the  curriculum.  To  limit  the  cost,  interviews  could  be 
conducted  in  a  few  MPT  billet  concentration  areas  such  as  Millington,  TN  and 
Washington,  DC.  These  two  areas  would  provide  the  largest  number  of 
graduates  and  supervisors  in  a  close  proximity.  (USN  2005b) 

4.  Additional  Value  Measuring  Methods 

Several  different  methods  for  measuring  value  of  education  and  training 
were  introduced  in  Chapter  II.  Many  of  these  studies  could  be  adapted  to  the 
MSA  curriculum. 

a.  Return  on  Investment 

Phillips  (1996)  introduced  return  on  investment  (ROI)  as  the  fifth 
level  of  evaluation  to  measure  the  value  of  education  and  training.  A  study 
conducted  by  Bowman  and  Mehay  (2004)  determined  the  ROI  of  graduate 
education  using  Navy  Surface  Warfare  Officers.  A  recommendation  for  future 
study  would  be  to  measure  ROI  for  the  MSA  curriculum.  Although  Bowman  and 
Mehay  (2004)  determined  the  return  on  investment  for  Navy-funded  graduate 
education,  the  results  are  not  specific  to  the  MSA  community  and  further  study 
might  prove  valuable. 

b.  Determining  Costs  of  the  MSA  Curriculum 

An  additional  recommendation  is  to  determine  the  cost  of  the  MSA 
curriculum.  Eckardt  (1997)  developed  a  series  of  equations  to  determine  the 

cost  of  a  degree  program  and  the  marginal  cost  of  enrolling  additional  students  in 
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the  program  at  NPS.  The  values  for  both  the  cost  per  student  and  the  marginal 
cost  differ  by  curricula.  (Eckardt  1997,  p.84)  The  equations  Eckardt  established 
were  based  on  the  Financial  Management  curriculum  but  could  be  adapted  for 
the  MSA  curriculum.  Cost  estimates  will  be  needed  to  determine  the  financial 
return  on  investment. 

c.  Knowledge  Value  Added 

Housel  and  Bell  (2001)  introduced  the  Knowledge  Value  Added 
(KVA)  approach  to  measure  the  value  of  knowledge  in  an  organization. 
Branstetter  (2002)  applied  the  concept  to  determine  the  value  to  the  Marine 
Corps  of  the  knowledge  learned  through  the  Information  System  Technology 
(1ST)  Curriculum.  A  recommendation  for  future  study  would  be  to  apply  the  same 
techniques  to  the  MSA  curriculum.  Conducting  a  KVA  analysis  for  the  MSA 
curriculum  would  yield  standardized  units  of  knowledge  for  comparison  in 
different  billets.  A  comparison  of  costs  could  then  be  conducted  between  the 
MSA  curriculum,  other  fully-funded  graduate  education  programs,  and  partially- 
funded  or  off-duty  graduate  education  programs  (via,  for  example,  tuition 
assistance)  using  standardized  units  for  knowledge  and  skills. 

5.  Related  Topic  Research 

The  final  recommendation  for  additional  research  is  to  develop  similar 
surveys  for  other  curricula  at  NPS.  Although  the  objective  of  this  thesis  was  to 
develop  instruments  to  be  used  for  the  MSA  curriculum,  the  surveys  developed  in 
this  thesis  could  easily  be  adapted  for  other  curricula.  An  initial  place  to  begin 
would  be  to  apply  the  surveys  to  other  MPT-related  curricula  such  as  Operations 
Analysis  and  the  Human  Systems  Integration  programs  at  NPS.  The  surveys 
developed  would  be  ideal  for  both  of  these  curricula  because  the  1200 
community  sends  officers  to  both  of  these  curricula  and  its  graduates  are  often 
sent  to  Manpower,  Personnel,  and  Training  billets. 

With  a  combination  of  the  two  surveys  and  these  different  methods,  the 
value  of  the  MSA  curriculum  could  be  determined  from  all  aspects  of  evaluation. 
The  graduate  survey  provides  the  perceived  value  of  the  curriculum,  Kirkpatrick’s 
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(1971)  reaction,  or  first  level  of  evaluation.  The  supervisor  survey  provides  some 
insight  on  the  learning  or  behavior  levels  at  the  second  and  third-levels  of 
evaluation.  Looking  at  retention  and  promotion  benefits  of  the  curriculum 
provides  an  evaluation  from  the  results  based,  or  fourth  level  of  Kirkpatrick’s 
(1971)  evaluation.  A  return-on-investment  analysis  of  the  curriculum 
encompasses  Phillips’  (1996)  fifth  level  of  evaluation.  Using  the  Knowledge 
Value  Added  approach  measures  the  value  of  the  knowledge  to  the  organization. 
Conducting  a  combination  of  these  different  methods  would  yield  a  more 
thorough  understanding  of  the  value  of  the  curriculum. 
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APPENDIX  A:  GRADUATE  SURVEY 


You  have  been  identified  as  having  attended  the  Naval  Postgraduate 
School  (NPS)  in  the  Manpower  Systems  Analysis  (MSA)  Curriculum.  Upon 
completion,  you  received  a  3130P-subspecialty,  as  a  Manpower  Systems 
Analyst.  Upon  completion  of  your  follow-on  tour,  your  subspecialty  becomes  a 
3130  Q. 

Thank  you  for  taking  the  time  to  complete  the  following  survey  about  your 
experience  as  a  student  in  the  Manpower  Systems  Analysis  (MSA)  Curriculum  at 
the  Naval  Postgraduate  School  (NPS). 

The  goal  of  this  survey  is  to  measure  the  perceived  value  of  the  MSA 
Curriculum  and  whether  the  curriculum  provided  you  a  foundation  for  your  follow- 
on  billets.  We  appreciate  any  insight  you  can  share  that  will  enable  us  to 
improve  the  quality  of  the  MSA  curriculum,  ensure  its  continued  relevancy,  and 
further  improve  the  reputation  of  your  alma  mater.  Thank  you  in  advance  for 
your  participation. 

The  survey  is  entirely  anonymous.  The  demographic  questions  are  for 
statistical  analysis  only. 

1 .  When  did  you  graduate  from  NPS  (Month  &  Year)?  _ 

2.  What  was  your  paygrade  when  you  completed  the  MSA  curriculum? 

a)  0-2  c)  0-4 

b)  0-3  d)  0-5 

3.  What  is  your  current  designator? _ 

4.  Did  you  laterally  transfer  into  this  community? 

a)  Yes. 

b)  No. 

If  yes,  what  was  your  previous  designator?  _ 

5.  Did  you  have  a  post-baccalaureate  degree  before  enrolling  at  NPS? 

a)  Yes 

b)  No 
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6.  Describe  your  application/selection  process  to  attend  NPS? 

a)  I  requested  to  attend 

b)  I  had  to  apply  and  be  selected  by  a  review  board 

c)  Sent  by  Detailer,  no  choice 

d)  Other 

If  other,  please  explain. _ 


7.  Which  degree  did  you  receive  while  at  NPS  for  the  MSA  Curriculum? 

a)  Master  of  Science  in  Management  (MSM) 

b)  Master  of  Business  Administration  (MBA) 

8.  After  receiving  your  MSA  degree,  have  you  filled  or  are  you  currently  filling  a  3130  P- 
or  Q-coded  billet?  If  so,  for  how  long  were  you  or  have  you  been  in  that  billet  (in 
closest  month)? 

a.  3130  P  No  Yes  For  how  long? 

b.  3130  Q  No  Yes  For  how  long? 

9.  If  your  current  or  previous  billet  is/was  not  P-or  Q-coded,  will  it  meet  your 
subspecialty  code  payback  requirement? 

a.  Yes 

b.  No 

c.  Unknown 

10.  How  would  you  rate  the  MSA  curriculum  in  preparing  you  for  your  manpower  follow- 
on  billets? 

a)  Very  Effective 

b)  Effective 

c)  Ineffective 

d)  Very  ineffective 

e)  N/A,  I  have  not  yet  filled  a  manpower  follow-on  billet 

If  you  answered  other  than  “A,”  what  was  missing  from  the  curriculum? _ 


1 1 .  How  did  you  select  your  thesis  topic? 

a)  Provided  by  MSA  Faculty 

b)  Sponsored  by  external  source  (please  indicate  who  your  sponsor  was) 

c)  Self-selected 

d)  Provided  by  other  NPS  Faculty/Staff 

e)  Current  or  Previous  Student  (Follow-on  thesis) 

f)  Other 

If  other,  please  explain  how  you  selected  your  topic: _ 
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12.  How  would  you  rate  the  overall  relevance  of  your  thesis  topic  to  your  jobs/career 
after  departing  NPS? 

a)  Very  Relevant 

b)  Relevant 

c)  Somewhat  relevant 

d)  Not  relevant  at  all 

If  you  answered  other  than  “A,”  please  explain  why.  _ 


13. To  what  extent  do  you  believe  interaction  with  DoD  military  and  civilian  students,  as 
well  as  international  students  in  the  MSA  curriculum,  added  any  value  to  your 
military  career? 

a)  To  a  very  great  extent 

b)  To  a  great  extent 

c)  To  a  small  extent 

d)  To  no  extent 

If  you  answered  other  than  “A,”  please  explain  why.  _ 


14. To  what  extent  was  the  curriculum  effective  in  covering  current  military  manpower 
issues  (current  at  the  time  you  were  enrolled)? 

a)  To  a  very  great  extent 

b)  To  a  great  extent 

c)  To  a  small  extent 

d)  To  no  extent 

If  you  answered  other  than  “A,”  please  explain  why.  _ 


15.  How  did  the  MSA  curriculum  at  NPS  improve  your  skills  in  quantitative  Data  Analysis 
and  in  the  application  of  analytical  tools? 

a)  To  a  very  great  extent 

b)  To  a  great  extent 

c)  To  a  small  extent 

d)  To  no  extent 

If  you  answered  other  than  “A,”  please  explain  why.  _ 


16.  How  did  the  MSA  curriculum  at  NPS  improve  your  knowledge  in  Manpower  Systems 
Analysis  Fundamental  Concepts  (Manpower,  Personnel  &  Training)? 

a)  To  a  very  great  extent 

b)  To  a  great  extent 

c)  To  a  small  extent 

d)  To  no  extent 

If  you  answered  other  than  “A,”  please  explain  why.  _ 
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17.  How  did  the  MSA  curriculum  at  NPS  improve  your  knowledge  in  Manpower  Systems 
Analysis  Policy  Analysis  (identifying  strengths  and  weaknesses  of  proposed  policies 
and  identifying  alternatives)? 

a)  To  a  very  great  extent 

b)  To  a  great  extent 

c)  To  a  small  extent 

d)  To  no  extent 

If  you  answered  other  than  “A,”  please  explain  why.  _ 


18.  If  you  received  an  MBA  from  this  program,  would  you  still  have  entered  the 
curriculum  if  the  Masters  of  Science  were  the  only  choice  (assuming  you  had  a 
choice)? 

a)  Yes 

b)  No 

c)  N/A,  I  received  a  Master  of  Science  (MS)  degree. 

If  no,  please  explain  why.  _ 


19.  Would  you  recommend  attendance  at  NPS  in  the  MSA  curriculum  to  others? 

a)  Yes 

b)  No 

If  no,  please  explain  why.  _ 


20.  As  an  Alumnus,  what  have  you  found  the  most  valuable  about  your  MSA  experience 
as  it  relates  to  your  Navy  Career? _ 


21  .Are  there  any  changes  or  additions  you  would  recommend  for  the  MSA  Curriculum 
that  you  have  not  yet  mentioned? _ 


22.  Please  include  any  other  remarks  you  would  like  to  make  about  the  MSA  curriculum 
or  your  experience  at  NPS. _ 
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APPENDIX  B:  IMMEDIATE  SUPERVISOR  SURVEY 


You  have  been  identified  as  having  administrative/operational  responsibility  of  at 
least  one  billet  that  requires  a  graduate  degree  (3130P)  or  graduate  degree  and  follow- 
on  tour  experience  (3130Q)  as  a  Manpower  Systems  Analyst. 

We  would  appreciate  your  taking  the  time  to  answer  the  following  survey  about 
your  Manpower  Systems  Analysis  (MSA)  graduate(s)  from  the  Naval  Postgraduate 
School  (NPS). 

The  goal  of  this  survey  is  to  measure  the  perceived  value  of  the  MSA  Curriculum. 
We  appreciate  any  insight  you  can  share  that  will  enable  us  to  improve  the  quality  of  the 
MSA  curriculum,  ensuring  its  continued  relevancy  to  meet  the  needs  of  the  Navy. 

Thank  you  in  advance  for  your  participation. 

The  survey  is  entirely  anonymous.  The  demographic  questions  are  for  statistical 
analysis  only. 

1 .  To  what  officer  category  do  you  belong? 

a)  Unrestricted  Line 

b)  Restricted  Line 

c)  Staff  Corps 

2.  What  is  your  paygrade? 


a) 

0-3 

e)  0-7 

') 

GS-13 

b) 

0-4 

f)  GS-10 

j) 

GS-14 

c) 

0-5 

g)  GS-11 

k) 

GS-15 

d) 

0-6 

h)  GS-12 

1) 

SES 

3.  If  you  have  the  Manpower  Systems  Analyst  31 30-subspecialty,  which  level 
subspecialty  do  you  have? 

a)  3130  P  (Subspecialist)  d)  3130  R  (Significant  Experience 

b)  3130  Q  (Proven  Subspecialist)  Proven  Subspecialist) 

c)  3130  S  (Significant  Experience)  e)  N/A 

4.  Please  select  the  Navy  Officer  Billet  Classification  groups  of  the  3130-coded  officers 
you  supervise.  (Select  all  that  apply). 

Health  Care  Services  Field: 

a)  Health  Services  Management  Group 

b)  Health  care  services  Group 

c)  Nursing  Group 
Personnel  Field: 

d)  Recruitment  and  Selection  Group 

e)  Classification  and  Distribution 
Group 

f)  General  Training  Group 

g)  Human  Resource  Management 
Group 


h)  Performance  Group 

i)  Welfare  Group 

j)  General  Group 

k)  Other- not  listed 

(Please  list) _ 


d)  Civilian 

e)  Other  (please  specify) 
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5.  How  many  3130  P-or  Q-subspecialty  officers  do  you  have  working  for  you? 

6.  Are  there  any  billets  under  your  direction  that  are  not  3130  P-or  Q-coded  that 
you  feel  should  be? 

a)  Yes 

b)  No 

If  you  answered  yes,  please  explain  why  it  should  be  a  P-or  Q-coded  billet. 


7.  Compared  to  a  Q-coded  (proven )-subspecialist,  at  what  proficiency  level 
would  you  say  your  MSA  graduate  was  upon  arrival? 


0% — | — 20%-— 

|— -40%-— | — 60% — | — 80% — 

-| — 100% 

About  how  many  months  does  it  usually  take  for  the  MSA  graduate  to  reach 
the  proficiency  of  a  3130Q-subspecialist  (proven)? 

a.  0-3  mo. 

e.  13-15  mo. 

i.  25-27  mo. 

b.  4-6  mo. 

f.  16-18  mo. 

j.  28-30  mo. 

c.  7-9  mo. 

g.  19-21  mo. 

k.  31-33  mo. 

d.  10-12  mo. 

h.  22-24  mo. 

1.  34-36  mo 

9.  Have  the  3130P-subspecialty  officers  who  have  worked  for  you  in  the  past,  or 
are  working  for  you  now,  arrived  with  the  skills  necessary  to  enable  them  to 
perform  their  job  sufficiently? 

a.  Yes 

b.  No. 

If  not,  what  skills  are  the  MSA  graduates  missing? _ 


10.  In  your  current  or  previous  supervisory  positions,  how  would  you  compare  the 
job  performance  of  MSA  graduates  with  other  officers  who  hold  a  master’s 
degree? 

a.  Much  better  prepared  than  other  graduates 

b.  Somewhat  better  prepared  than  other  graduates 

c.  About  the  same  as  other  graduates 

d.  Somewhat  less  prepared  than  other  graduates 

e.  Much  less  prepared  than  other  graduates 
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1 1 .  In  your  current  or  previous  supervisory  positions,  how  would  you  compare  the 
job  performance  of  MSA  graduates  with  other  officers  who  don’t  have  a 
master’s  degree? 

a.  Much  better  prepared  than  non-graduates 

b.  Somewhat  better  prepared  than  non-graduates 

c.  About  the  same  as  non-graduates 

d.  Somewhat  less  prepared  than  non-graduates 

e.  Much  less  prepared  than  non-graduates 

12.  Would  you  request  an  MSA  graduate  from  NPS  for  any  non-subspecialty- 
coded  billets  over  other  master’s  graduates  if  you  could? 

a.  Yes,  I  would  prefer  an  MSA  graduate  degree 

b.  I’m  neutral.  I  will  take  any  graduate  degree 

c.  No,  I  would  prefer  another  graduate  over  an  MSA  graduate  degree 

d.  N/A,  graduate  education  is  not  needed. 

If  no,  please  explain  why  and  who  you  would  prefer: _ 


13.  Is  there  anything  about  your  NPS  MSA  graduate  that  stands  out  from  other 
officers  you  have  worked  with? 
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